Commit Graph

27672 Commits

Author SHA1 Message Date
Martijn van Groningen 6945d7b046
test: Stop using the `mapping.single_type` setting in percolator tests.
Closes #24958
2017-05-31 09:11:33 +02:00
Adrien Grand 36a180ec20 Eliminate array access in tight loops when profiling is enabled. (#24959)
This makes profiling classes acquire a timer up-front that can be then reused
across all calls, in order to save bound checks for methods that are called in
tight loops.
2017-05-31 09:11:00 +02:00
Adrien Grand 71264c6239 PatternAnalyzer should lowercase wildcard queries when `lowercase` is true. (#24967) 2017-05-31 09:09:53 +02:00
Ryan Ernst 7c1211d2ed Scripting: Add StatefulFactoryType as optional intermediate factory in script contexts (#24974)
ScriptContexts currently understand a FactoryType that can produce
instances of the script InstanceType. However, for search scripts, this
does not work as we have the concept of LeafSearchScript that is created
per lucene segment. This commit effectively renames the existing
SearchScript class into SearchScript.LeafFactory, which is a new,
optional, class that can be defined within a ScriptContext.
LeafSearchScript is effectively renamed back into SearchScript. This
change allows the model of stateless factory -> stateful factory ->
script instance to continue, but in a generic way that any script
context may take advantage of.

relates #20426
2017-05-30 16:32:14 -07:00
Jack Conradson 04daac2243 Make Painless Compiler Use an Instance Per Context (#24972)
Allows for easier management of compilation of individual interfaces on a per script context basis.
2017-05-30 14:30:48 -07:00
Jason Tedor ac94253dce Clarify acquiring index shard permit
In previous work, we refactored the delay mechanism in index shard
operation permits to allow for async delaying of acquisition. This
refactoring made explicit when permit acquisition is disabled whereas
previously we were relying on an implicit condition, namely that all
permits were acquired by the thread trying to delay acquisition. When
using the implicit mechanism, we tried to acquire a permit and if this
failed, we returned a null releasable as an indication that our
operation should be queued. Yet, now we know when we are delayed and we
should not even try to acquire a permit. If we try to acquire a permit
and one is not available, we know that we are not delayed, and so
acquisition should be successful. If it is not successful, something is
deeply wrong. This commit takes advantage of this refactoring to
simplify the internal implementation.

Relates #24971
2017-05-30 16:22:17 -04:00
Jason Tedor 9b6b4ffe8e Set number of processes in systemd unit file
This commit sets the number of processes in the systemd unit file for
Elasticsearch to meet the bootstrap checks.

Relates #24970
2017-05-30 15:34:20 -04:00
Jason Tedor b28141a990 Fill gaps on primary promotion
When a primary is promoted, it could have gaps in its history due to
concurrency and in-flight operations when it was serving as a
replica. This commit fills the gaps in the history of the promoted shard
after all operations from the previous term have drained, and future
operations are blocked. This commit does not handle replicating the
no-ops that fill the gaps to any remaining replicas, that is the
responsibility of the primary/replica sync that we are laying the ground
work for.

Relates #24945
2017-05-30 13:19:44 -04:00
Jim Ferenczi ce7195d81a Terms aggregation should remap global ordinal buckets when a sub-aggregator is used to sort the terms (#24941)
`terms` aggregations at the root level use the `global_ordinals` execution hint by default.
When all sub-aggregators can be run in `breadth_first` mode the collected buckets for these sub-aggs are dense (remapped after the initial pruning).
But if a sub-aggregator is not deferrable and needs to collect all buckets before pruning we don't remap global ords and the aggregator needs to deal with sparse buckets.
Most (if not all) aggregators expect dense buckets and uses this information to allocate memories.
This change forces the remap of the global ordinals but only when there is at least one sub-aggregator that cannot be deferred.

Relates #24788
2017-05-30 19:13:07 +02:00
Tal Levy 2a6e6866bd Fix floating-point error when DateProcessor parses UNIX (#24947)
DateProcessor's DateFormat UNIX format parser resulted in
a floating point rounding error when parsing certain stringed
epoch times. Now Double.parseDouble is used, preserving the
intented input.
2017-05-30 09:42:26 -07:00
Jack Conradson 5bcae914d9 Make PainlessScript an interface (#24966)
Allows more flexibility for the specified script context interface if we want to allow script contexts to specify an abstract class instead.
2017-05-30 09:03:46 -07:00
Jason Tedor ddbc4687f6 Introduce clean transition on primary promotion
This commit introduces a clean transition from the old primary term to
the new primary term when a replica is promoted primary. To accomplish
this, we delay all operations before incrementing the primary term. The
delay is guaranteed to be in place before we increment the term, and
then all operations that are delayed are executed after the delay is
removed which asynchronously happens on another thread. This thread does
not progress until in-flight operations that were executing are
completed, and after these operations drain, the delayed operations
re-acquire permits and are executed.

Relates #24925
2017-05-30 11:39:36 -04:00
Jason Tedor 15fc71249c Fix typo in comment in ReplicationOperation.java
Within two lines of each other appears "fallthrough" and "fall through",
both typed by the same person who should have been paying better
attention and only one of these is correct and the inconsistency is
bothersome. This commit fixes the errant one.
2017-05-30 11:16:32 -04:00
Lee Hinman 0b3be42c10 Prevent Index & Delete request primaryTerm getter/setter, setShardId setter 2017-05-30 08:48:35 -06:00
Nik Everett 6d9ce957d4 Drop name from TokenizerFactory (#24869)
Drops `TokenizerFactory#name`, replacing it with
`CustomAnalyzer#getTokenizerName` which is much better targeted at
its single use case inside the analysis API.

Drops a test that I would have had to refactor which is duplicated by
`AnalysisModuleTests`.

To keep this change from blowing up in size I've left two mostly
mechanical changes to be done in followups:
1. `TokenizerFactory` can now be entirely dropped and replaced with
`Supplier<Tokenizer>`.
2. `AbstractTokenizerFactory`'s ctor still takes a `String` parameter
where the name once was.
2017-05-30 10:39:22 -04:00
Zachary Tong b8d7b83f8e Correctly set doc_count when MovAvg "predicts" values on existing buckets (#24892)
If the bucket already exists, due to non-overlapping series or missing data, the
MovAvg creates a merged bucket with the existing aggs + the new prediction.  This
fixes a small bug where the doc_count was not being set correctly.

Relates to #24327
2017-05-30 10:36:22 -04:00
Jason Tedor 9957bdf0ad Handle primary failure handling replica response
Today if the primary throws an exception while handling the replica
response (e.g., because it is already closed while updating the local
checkpoint for the replica), or because of a bug that causes an
exception to be thrown in the replica operation listener, this exception
is caught by the underlying transport handler plumbing and is translated
into a response handler failure transport exception that is passed to
the onFailure method of the replica operation listener. This causes the
primary to turn around and fail the replica which is a disastrous and
incorrect outcome as there's nothing wrong with the replica, it is the
primary that is broken and deserves a paddlin'. This commit handles this
situation by failing the primary.

Relates #24926
2017-05-30 10:05:11 -04:00
David Cho-Lerat 491dc1186a Add missing word to terms-query.asciidoc (#24960) 2017-05-30 09:42:07 -04:00
David Cho-Lerat c939bcb7f5 Correct some spelling in match-phrase-prefix docs (#24956) 2017-05-30 09:02:01 -04:00
Boaz Leskes efcbfb7c32 testConcurrentWriteViewsAndSnapshot shouldn't flush concurrently
Fixes #24933
2017-05-30 14:22:23 +02:00
Jim Ferenczi fc35d51c3c [TEST] Fix FieldSortIT failures 2017-05-30 11:06:54 +02:00
Tanguy Leroux eea010b408 Add doc_count to ParsedMatrixStats (#24952)
This commit adds support in ParsedMatrixStats for parsing the doc_count
field.

Related to #24776
2017-05-30 10:16:08 +02:00
Tanguy Leroux 28d97df67c Add document count to Matrix Stats aggregation response (#24776)
This commit adds a `doc_count` field to the response body of Matrix
Stats aggregation. It exposes the number of documents involved in
 the computation of statistics, a value that can already be retrieved using
  the method MatrixStats.getDocCount() in the Java API.
2017-05-30 09:39:41 +02:00
Jim Ferenczi 628dabd663 Fix script field sort returning Double.MAX_VALUE for all documents (#24942)
This change fixes the script field sort when the returned type is a number.

Closes #24940
2017-05-30 09:23:04 +02:00
Jason Tedor b2abdd1174 Add a second refresh to concurrent relocation test
This commit adds a second refresh to the concurrent relocation
test. This is necessary as the first refresh might have brought back a
local checkpoint for a shard that a newly relocated primary became aware
of but did not yet receive a local checkpoint for that shard. When that
local checkpoint arrives on the new primary, the global checkpoint could
advance again and so we need a second replication action to push that
global checkpoint back out to the replica. This is indeed a hack, and it
will eventually be removed.

Closes #24599
2017-05-29 11:37:27 -04:00
Simon Willnauer 4d423bf2ba Add a dummy_index to upgrade tests to ensure we recover fine with replicas (#24937)
We default to 0 replicas in the rolling restart scenario already to ensure
we test against worst case. Yet, this adds a dummy index to ensure we also
recover and index with replicas just fine.
2017-05-29 17:36:44 +02:00
Nik Everett bc4df47a76 Rework bwc snapshot projects to build up to two bwc versions (#24870)
Removes the `distribution:bwc` project in favor of
`distribution:bwc-release-snapshot` and
`distribution:bwc-stable-snapshot`.
`distribution:bwc-release-snapshot` builds a snapshot of the
latest release branch (5.4 now) if needed for backwards
compatibility. `distribution:bwc-stable-snapshot` builds a
snapshot of the latest stable branch (5.x now) if needed for
backwards compatibility.
2017-05-29 10:22:32 -04:00
Boaz Leskes dfdf496c1a Move the IndexDeletionPolicy to be engine internal (#24930)
The `IndexDeletionPolicy` is currently instantiated by `IndexShard` and is then passed through to the engine as a parameter. That's a shame as it is really just an implementation detail and the engine already has a method to acquire a commit.

This is preparing for a follow up PR that will we connect the index deletion policy with a new translog deletion policy.

Relates to #10708
2017-05-29 15:56:30 +02:00
Christoph Büscher 5741005812 [Tests] Harden InternalExtendedStatsTests (#24934)
The order in which double values are added in Java can give different results,
so in testing the sum and sumOfSquares we need to allow some delta for testing
equality. The difference can be larger for large sum values, so we should
account for this by making the delta in the assertion depend on the values
magnitude.

Closes #24931
2017-05-29 15:45:51 +02:00
propulkit 25516868fe TCorrecting api name (#24924)
As per REST request signature for reroute, API has no underscore.
2017-05-29 13:58:31 +02:00
Luca Cavanna c120f8af89 Add search method to high level REST client (#24796)
Relates to #23331
2017-05-29 12:47:34 +02:00
Luca Cavanna 12654384a5 Add fromXContent method to ClearScrollResponse (#24909)
ClearScrollResponse can print out its content into an XContentBuilder as it implements ToXContentObject. This PR add a fromXContent method to it so that we are able to recreate the response object when parsing the response back. This will be used in the high level REST client.
2017-05-29 11:00:54 +02:00
Luca Cavanna ea00d343ae ClearScrollRequest to implement ToXContentObject (#24907)
ClearScrollRequest can be created from a request body, but it doesn't support the opposite, meaning printing out its content to an XContentBuilder. This is useful to the high level REST client and allows for better testing of what we parse.

Moved parsing method from RestClearScrollAction to ClearScrollRequest so that fromXContent and toXContent sit close to each other. Added unit tests to verify that body parameters override query_string parameters when both present (there is already a yaml test for this but unit test is even better)
2017-05-29 11:00:20 +02:00
Luca Cavanna 7be5758e6d SearchScrollRequest to implement ToXContentObject (#24906)
SearchScrollRequest can be created from a request body, but it doesn't support the opposite, meaning printing out its content to an XContentBuilder. This is useful to the high level REST client and allows for better testing of what we parse.

Moved parsing method from RestSearchScrollAction to SearchScrollRequest so that fromXContent and toXContent sit close to each other. Added unit tests to verify that body parameters override query_string parameters when both present (there is already a yaml test for this but unit test is even better)
2017-05-29 10:56:35 +02:00
mashudong 678730107c Fix bug in weight computation for query cache
When proportioning the shared RAM bytes across the shards of the query
cache, there's a computation that shares these bytes according to the
relative size of the shard cache to the total size of all the shard
caches. This computation had a bug where integer division was performed
instead which leads to this computation often being zero. This commit
fixes this bug by casting the numerator to a double before doing the
division so that double division is performed.

Relates #24856
2017-05-28 13:36:47 -04:00
Jason Tedor d1318e4d23 Fix Lucene version expectation
The Lucene version expectation in the verify Lucene version test is
backwards, mixing up the expected and actual values. This commit
reorders them to fix this issue.
2017-05-27 18:31:44 -04:00
Jason Tedor 09dd03e19f Verify Lucene version constants
The Lucene version constants for 5.4.1 and 5.5.0 are wrong, they are
listed as 6.5.0 instead of 6.5.1. This commit fixes these issues, and
adds a test to ensure that this does not happen again.

Relates #24923
2017-05-27 15:46:16 -04:00
Jason Tedor 3448028af7 Avoid double decrement on current query counter
This commit fixes a double decrement bug on the current query
counter. The double decrement arises in a situation when the fetch phase
is inlined for a query that is only touching one shard. After the query
phase succeeds we decrement the current query counter. If the fetch
phase ultimately fails, an exception is thrown and we decrement the
current query counter again in the catch block. We also add assertions
that all current stats counters remain non-negative at all
times.

Relates #24922
2017-05-27 15:40:37 -04:00
Nik Everett 5da8ce8318 Remove the need for _UNRELEASED suffix in versions (#24798)
Removes the need for the `_UNRELEASED` suffix on versions by detecting if a version should be unreleased or not based on the versions around it. This should make it simpler to automate the task of adding a new version label.
2017-05-26 18:36:32 -04:00
Lee Hinman 23fb36cc87 Adjust available and free bytes to be non-negative on huge FSes
In #23093 we made a change so that total bytes for a filesystem would not be a
negative value when the total bytes were > Long.MAX_VALUE.

This fixes #24453 which had a related issue where `available` and `free` bytes
could also be so large that they were negative. These will now return
`Long.MAX_VALUE` for the bytes if the JDK returns a negative value.
2017-05-26 12:59:53 -06:00
Nik Everett e072cc7770 Begin replacing static index tests with full restart tests (#24846)
These tests spin up two nodes of an older version of Elasticsearch,
create some stuff, shut down the nodes, start the current version,
and verify that the created stuff works.

You can run `gradle qa:full-cluster-restart:check` to run these
tests against the head of the previous branch of Elasticsearch
(5.x for master, 5.4 for 5.x, etc) or you can run
`gradle qa:full-cluster-restart:bwcTest` to run this test against
all "index compatible" versions, one after the other. For master
this is every released version in the 5.x.y version *and* the tip
of the 5.x branch.

I'd love to add more to these tests in the future but these
currently just cover the functionality of the `create_bwc_index.py`
script and start to cover the assertions in the
`OldIndexBackwardsCompatibilityIT` test.
2017-05-26 14:07:48 -04:00
Jason Tedor bb63577f14 Fix plugin docs for using custom config dir
This commit fixes an issue with the plugin docs incorrectly specifying
how to set a custom configuration directory. The correct way is to use
the environment variable CONF_DIR.
2017-05-26 13:40:35 -04:00
Clinton Gormley 0656d0236b Update context-suggest.asciidoc
Removed incorrect parameter
2017-05-26 17:41:40 +02:00
Jim Ferenczi 47cf7825dd Move BWC version to 5.5 after backport
Relates to #24517
2017-05-26 14:57:07 +02:00
Jim Ferenczi 9ef414fead Merge branch 'mattweber-multiple_collapse_inner_hits' 2017-05-26 13:28:08 +02:00
Matt Weber 601a61a91c Support Multiple Collapse Inner Hits
Support multiple named inner hits on a field collapsing
request.
2017-05-26 13:23:57 +02:00
Ryan Ernst 74e031e842 Scripting: Rename CompiledType to FactoryType in ScriptContext (#24897)
This commit renames the concept of the "compiled type" to a "factory
type", along with all implementations of this class to be named Factory.
This brings it inline with the classes purpose.
2017-05-26 00:02:54 -07:00
Ryan Ernst 8eab1fefa1 Scripting: Make contexts available to ScriptEngine construction (#24896)
This commit adds collection of all contexts to the parameters of
getScriptEngine. This will allow script engines like painless to
precache extra information about the contexts.
2017-05-25 16:55:47 -07:00
Jason Tedor 9a118691bf Mute index and relocate concurrently
This test is failing sporadically and for now we mute it as we have a
failure with additional logging that should hopefully enable us to
assess the situation.
2017-05-25 17:37:36 -04:00
Ryan Ernst 8dbf8a230e Build: Add back explicit exclusions and remove gradle exclusions (#24879)
When transitive dependencies are disable for a dependency, gradle adds a
wildcard exclusion to the generated pom. However, some external tools
like ivy have bugs with wildcards. This commit adds back the explicit
generation of transitive excludes, and removes the gradle generated
exclusions element from the pom.

closes #24490
2017-05-25 12:22:08 -07:00