8343 Commits

Author SHA1 Message Date
Jim Ferenczi
fc35d51c3c [TEST] Fix FieldSortIT failures 2017-05-30 11:06:54 +02:00
Jim Ferenczi
628dabd663 Fix script field sort returning Double.MAX_VALUE for all documents (#24942)
This change fixes the script field sort when the returned type is a number.

Closes #24940
2017-05-30 09:23:04 +02:00
Jason Tedor
b2abdd1174 Add a second refresh to concurrent relocation test
This commit adds a second refresh to the concurrent relocation
test. This is necessary as the first refresh might have brought back a
local checkpoint for a shard that a newly relocated primary became aware
of but did not yet receive a local checkpoint for that shard. When that
local checkpoint arrives on the new primary, the global checkpoint could
advance again and so we need a second replication action to push that
global checkpoint back out to the replica. This is indeed a hack, and it
will eventually be removed.

Closes #24599
2017-05-29 11:37:27 -04:00
Boaz Leskes
dfdf496c1a Move the IndexDeletionPolicy to be engine internal (#24930)
The `IndexDeletionPolicy` is currently instantiated by `IndexShard` and is then passed through to the engine as a parameter. That's a shame as it is really just an implementation detail and the engine already has a method to acquire a commit.

This is preparing for a follow up PR that will we connect the index deletion policy with a new translog deletion policy.

Relates to #10708
2017-05-29 15:56:30 +02:00
Christoph Büscher
5741005812 [Tests] Harden InternalExtendedStatsTests (#24934)
The order in which double values are added in Java can give different results,
so in testing the sum and sumOfSquares we need to allow some delta for testing
equality. The difference can be larger for large sum values, so we should
account for this by making the delta in the assertion depend on the values
magnitude.

Closes #24931
2017-05-29 15:45:51 +02:00
Luca Cavanna
12654384a5 Add fromXContent method to ClearScrollResponse (#24909)
ClearScrollResponse can print out its content into an XContentBuilder as it implements ToXContentObject. This PR add a fromXContent method to it so that we are able to recreate the response object when parsing the response back. This will be used in the high level REST client.
2017-05-29 11:00:54 +02:00
Luca Cavanna
ea00d343ae ClearScrollRequest to implement ToXContentObject (#24907)
ClearScrollRequest can be created from a request body, but it doesn't support the opposite, meaning printing out its content to an XContentBuilder. This is useful to the high level REST client and allows for better testing of what we parse.

Moved parsing method from RestClearScrollAction to ClearScrollRequest so that fromXContent and toXContent sit close to each other. Added unit tests to verify that body parameters override query_string parameters when both present (there is already a yaml test for this but unit test is even better)
2017-05-29 11:00:20 +02:00
Luca Cavanna
7be5758e6d SearchScrollRequest to implement ToXContentObject (#24906)
SearchScrollRequest can be created from a request body, but it doesn't support the opposite, meaning printing out its content to an XContentBuilder. This is useful to the high level REST client and allows for better testing of what we parse.

Moved parsing method from RestSearchScrollAction to SearchScrollRequest so that fromXContent and toXContent sit close to each other. Added unit tests to verify that body parameters override query_string parameters when both present (there is already a yaml test for this but unit test is even better)
2017-05-29 10:56:35 +02:00
mashudong
678730107c Fix bug in weight computation for query cache
When proportioning the shared RAM bytes across the shards of the query
cache, there's a computation that shares these bytes according to the
relative size of the shard cache to the total size of all the shard
caches. This computation had a bug where integer division was performed
instead which leads to this computation often being zero. This commit
fixes this bug by casting the numerator to a double before doing the
division so that double division is performed.

Relates #24856
2017-05-28 13:36:47 -04:00
Jason Tedor
09dd03e19f Verify Lucene version constants
The Lucene version constants for 5.4.1 and 5.5.0 are wrong, they are
listed as 6.5.0 instead of 6.5.1. This commit fixes these issues, and
adds a test to ensure that this does not happen again.

Relates #24923
2017-05-27 15:46:16 -04:00
Jason Tedor
3448028af7 Avoid double decrement on current query counter
This commit fixes a double decrement bug on the current query
counter. The double decrement arises in a situation when the fetch phase
is inlined for a query that is only touching one shard. After the query
phase succeeds we decrement the current query counter. If the fetch
phase ultimately fails, an exception is thrown and we decrement the
current query counter again in the catch block. We also add assertions
that all current stats counters remain non-negative at all
times.

Relates #24922
2017-05-27 15:40:37 -04:00
Nik Everett
5da8ce8318 Remove the need for _UNRELEASED suffix in versions (#24798)
Removes the need for the `_UNRELEASED` suffix on versions by detecting if a version should be unreleased or not based on the versions around it. This should make it simpler to automate the task of adding a new version label.
2017-05-26 18:36:32 -04:00
Lee Hinman
23fb36cc87 Adjust available and free bytes to be non-negative on huge FSes
In #23093 we made a change so that total bytes for a filesystem would not be a
negative value when the total bytes were > Long.MAX_VALUE.

This fixes #24453 which had a related issue where `available` and `free` bytes
could also be so large that they were negative. These will now return
`Long.MAX_VALUE` for the bytes if the JDK returns a negative value.
2017-05-26 12:59:53 -06:00
Nik Everett
e072cc7770 Begin replacing static index tests with full restart tests (#24846)
These tests spin up two nodes of an older version of Elasticsearch,
create some stuff, shut down the nodes, start the current version,
and verify that the created stuff works.

You can run `gradle qa:full-cluster-restart:check` to run these
tests against the head of the previous branch of Elasticsearch
(5.x for master, 5.4 for 5.x, etc) or you can run
`gradle qa:full-cluster-restart:bwcTest` to run this test against
all "index compatible" versions, one after the other. For master
this is every released version in the 5.x.y version *and* the tip
of the 5.x branch.

I'd love to add more to these tests in the future but these
currently just cover the functionality of the `create_bwc_index.py`
script and start to cover the assertions in the
`OldIndexBackwardsCompatibilityIT` test.
2017-05-26 14:07:48 -04:00
Jim Ferenczi
47cf7825dd Move BWC version to 5.5 after backport
Relates to #24517
2017-05-26 14:57:07 +02:00
Jim Ferenczi
9ef414fead Merge branch 'mattweber-multiple_collapse_inner_hits' 2017-05-26 13:28:08 +02:00
Matt Weber
601a61a91c Support Multiple Collapse Inner Hits
Support multiple named inner hits on a field collapsing
request.
2017-05-26 13:23:57 +02:00
Ryan Ernst
74e031e842 Scripting: Rename CompiledType to FactoryType in ScriptContext (#24897)
This commit renames the concept of the "compiled type" to a "factory
type", along with all implementations of this class to be named Factory.
This brings it inline with the classes purpose.
2017-05-26 00:02:54 -07:00
Ryan Ernst
8eab1fefa1 Scripting: Make contexts available to ScriptEngine construction (#24896)
This commit adds collection of all contexts to the parameters of
getScriptEngine. This will allow script engines like painless to
precache extra information about the contexts.
2017-05-25 16:55:47 -07:00
Jason Tedor
9a118691bf Mute index and relocate concurrently
This test is failing sporadically and for now we mute it as we have a
failure with additional logging that should hopefully enable us to
assess the situation.
2017-05-25 17:37:36 -04:00
Ryan Ernst
8aaea51a0a Scripting: Move context definitions to instance type classes (#24883)
This is a simple refactoring to move the context definitions into the
type that they use. While we have multiple context names for the same
class at the moment, this will eventually become one ScriptContext per
instance type, so the pattern of a static member on the interface called
CONTEXT can be used. This commit also moves the consolidated list of
contexts provided by core ES into ScriptModule.
2017-05-25 12:18:45 -07:00
Jay Modi
f60f79f361 Put mapping and index template requests do not need content type detection for 5.3.0+ (#24835)
This change cleans up some missed TODOs for content type detection on the source of put mapping and
put index template requests. In 5.3.0 and newer versions, the source is always JSON so the content
type detection is not needed. The TODOs were missed after the change was backported to 5.3.

Relates #24798
2017-05-25 12:51:44 -06:00
Jay Modi
fab6b00ca7 Add the ability to store objects with a ScrollContext (#24777)
This commit adds the ability to store and retrieve data that should be associated with a
ScrollContext. Additionally the ScrollContext was made final as we should only have a single
implementation of this concept.
2017-05-25 12:30:00 -06:00
Simon Willnauer
1325681a03 Fix error message if an incompatible node connects (#24884)
This message broken in recent refactoring, this commit also adds a
basic unit-test to ensure we maintain the correct version.
2017-05-25 15:02:43 +02:00
Ryan Ernst
7d03cff820 Scripting: Make ScriptEngine.compile generic on the script context (#24873)
This commit changes the compile method of ScriptEngine to be generic in
the same way it is on ScriptService. This moves the shim of handling the
two existing context classes into each script engine, so that each
engine can be worked on independently to convert to real handling of
contexts.
2017-05-24 20:06:32 -07:00
Ryan Ernst
5581a0b2f0 Scripting: Remove unnecessary generic type from ScriptContext (#24877)
When developing the new ScriptContext, the compiled type was original
generic, so that the instance type was also necessary. However, since
CompiledType is all that is used by the compile method signature, we
actually don't need the instance type to be generic. This commit removes
the InstanceType, and finds the Class for it through reflection on the
CompiledType method.
2017-05-24 19:20:49 -07:00
Ryan Ernst
1daacd97b0 Scripting: Add instance and compiled classes to script contexts (#24868)
This commit modifies the compile method of ScriptService to be context
aware. The ScriptContext is now a generic class which contains both the
instance type and compiled type for a script. Instance type may be
stateful (for example, pre loading field information for the index a
script will execute on, like in expressions), while the compiled type is
stateless and used to construct instance type instances. This change is
only a first step to cutover ScriptService to the new paradigm. It only
converts callers to the script service, and has a small shim to wrap
compilation from the script engines to support the current two fixed
instance types, SearchScript and ExecutableScript.
2017-05-24 14:29:02 -07:00
Ryan Ernst
0ddd219423 Scripting: Add default implementation of close() for ScriptEngine (#24851)
Since groovy was removed, we no longer have any ScriptEngines with
resources to release. We may want to keep the option open for a script
engine to close resources, but this would not be common. This commit
adds a default implementation to ScriptEngine for `close()` to reduce
the boiler plate that must be added for a ScriptEngine implementation.
2017-05-24 13:19:27 -07:00
Lee Hinman
7a6db074ee [TEST] Add test for retrying replica operations with real network
Related to #24745
2017-05-24 13:31:30 -06:00
Jason Tedor
0499849b1b Increase logging in concurrent relocation test
This commit increases the logging level on the index and relocate
concurrently test to obtain some insight into the global checkpoint
moving backwards.
2017-05-24 11:51:00 -04:00
Boaz Leskes
b58f198d94 NoMasterNodeIT shouldn't try to validate the length of a timeout
The current log tries make sure we waited some (but not too long). This is unpredictable and fails all the time. This commit removes all of it and just make sure that we throw the right exceptions after timing out.

Fixes #24369
2017-05-24 16:15:15 +02:00
Simon Willnauer
aa0b9f9b39 Remote duplicate test from RemoteClusterServiceTests
The test `testEnsureConnected` was accidentially introduces during #24845
2017-05-24 16:04:25 +02:00
Ali Beyad
877f2d671b [TEST] adds TRACE logging to org.elasticsearch.snapshots for the
SharedClusterSnapshotRestoreIT#testDataFileCorruptionDuringRestore test
2017-05-24 09:50:17 -04:00
Ali Beyad
68f03b88c4 [TEST] increases testDataFileCorruptionDuringRestore logging to TRACE to
help debug sporadic tests failures.
2017-05-24 09:38:54 -04:00
markharwood
b7197f5e21 SignificantText aggregation - like significant_terms, but for text (#24432)
* SignificantText aggregation - like significant_terms but doesn’t require fielddata=true, recommended used with `sampler` agg to limit expense of tokenizing docs and takes optional `filter_duplicate_text`:true setting to avoid stats skew from repeated sections of text in search results.

Closes #23674
2017-05-24 13:46:43 +01:00
Yannick Welsch
b5adb3cce9 Remove AlreadyExpiredException (#24857)
This is a relict from the TTL functionality that has been removed in #21670
2017-05-24 13:49:02 +02:00
Boaz Leskes
6bc5b1dbcd Guarantee that translog generations are seqNo conflict free (#24825)
With #24779 in place, we can now guaranteed that a single translog generation file will never have a sequence number conflict that needs to be resolved by looking at primary terms. These conflicts can a occur when a replica contains an operation which isn't part of the history of a newly promoted primary. That primary can then assign a different operation to the same slot and replicate it to the replica.

PS. Knowing that each generation file is conflict free will simplifying repairing these conflicts when we read from the translog.

PPS. This PR also fixes some bugs in the piping of primary terms in the bulk shard action. These bugs are a result of the legacy of IndexRequest/DeleteRequest being a ReplicationRequest. We need to change that as a follow up.

Relates to #10708
2017-05-24 13:26:39 +02:00
Simon Willnauer
ac6a6d6fe8 Ensure remote cluster is connected before fetching _field_caps (#24845)
If a cluster disconnects and comes back up we should ensure that
we connected to the cluster before we fire the requests.

Closes #24763
2017-05-24 13:15:28 +02:00
Ryan Ernst
bf49d37ab3 Test: Convert test script engine impls to use MockScriptEngine (#24854)
This commit cleans up tests which currently use custom script engine
implementations, converting them to use a MockScriptEngine with script
functions provided by the tests. It also creates a common set of metric
scripts which were copied across a couple metric agg tests.
2017-05-23 20:34:12 -07:00
Jim Ferenczi
4707377cea Move InnerHitBuilder queries BWC version to 5.5 after the backport
Relates #24676
2017-05-23 22:41:39 +02:00
Christoph Büscher
24a8ba5ca8 Use ParseField constants in ParsedGeoBounds (#24849) 2017-05-23 20:15:40 +02:00
Lee Hinman
82a369737b [TEST] Add additional assert logging 2017-05-23 10:37:40 -06:00
markharwood
c71ae3519f Search: Fairer balancing when routing searches by session ID (#24671)
A user reported uneven balancing of load on nodes handling search requests from Kibana which supplies a session ID in a routing preference. Each shardId was selecting the same node for a given session ID because one data node had all primaries and the other data node held all replicas after cluster startup.
This change counteracts the tendency to opt for the same node given the same user-supplied preference by incorporating shard ID in the hash of the preference key. This will help randomise node choices across shards.

Closes #24642
2017-05-23 14:43:44 +00:00
Boaz Leskes
b2ccb6b0a8 simplify flow/stop control in testConcurrentWriteViewsAndSnapshot
Thread starvation of the control thread could cause the writer to keep on generating ops and make the test go out of control (OOM).
2017-05-23 15:40:53 +02:00
Jason Tedor
6ec485d30d Suppress inspections on assertions helper
This commit adds comments to org.elasticsearch.Assertions that disables
IntelliJ from complaining about using assert with side-effects, and
using constant conditions there as the side-effect with a constant
condition is intentionally employed.
2017-05-23 09:34:51 -04:00
Jason Tedor
c179c6a4c9 Add assertions enabled helper
Today in the code base we have lots of ugly code blocks like:

  boolean assertionsEnabled = false;
  assert assertionsEnabled = true;
  if (assertionsEnabled) {
    // something
  }

These are a nuisance. Instead, we can do this in exactly one place and
replace these blocks with

  if (Assertions.ENABLED) {
    // something
  }

The cool thing here is that since this is a static final field, the JIT
can optimize away the check at runtime if assertions are disabled.

Relates #24834
2017-05-23 08:22:18 -04:00
Jim Ferenczi
9087803cd9 Add the ability to define custom inner hit sub context builder (#24676)
This commit moves the handling of nested and parent/child inner hits to specialized classes that can be defined outside of ES core.
InnerHitBuilderContext is now used by the parent query (nested or hasChild, ...) to build the sub context from the InnerHitBuilder definition.
BWC is also ensured so that nodes in previous versions can still send/receive inner hits to/from this version.

Relates #20257
2017-05-23 13:06:22 +02:00
Boaz Leskes
a5fffa2988 relax the assertion about the existence of MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID in the user commit data
Since #24149 we started storing the maxUnsafeAutoIdTimestamp in the lucene commit. The assertions in that PR were built on the assumption that recovering a primary always ends up committing lucene. That is sadly [not true](a72eaa8e0f/core/src/main/java/org/elasticsearch/index/engine/InternalEngine.java (L313)).

Closes #24809
2017-05-23 09:08:30 +02:00
Ali Beyad
d3c95b8d9d Removes the 5.3.3 version constant (#24833)
After releasing 5.3.2, the 5.3.3 version constant was created.  However,
this causes issues for the rolling upgrade tests, which expect to have
all older versions artifacts published and no point releases created off
of the older versions (older meaning more than one version behind the
current version).  This commit removes the 5.3.3 version constant,
assuming we will not need it anywhere.
2017-05-22 21:53:16 -04:00
Jack Conradson
8887bcc4c6 Fix settings names for script.allowed_types and script.allowed_contexts. (#24831)
Fixes #24830
2017-05-22 15:08:45 -07:00