Commit Graph

8638 Commits

Author SHA1 Message Date
Daniel Mitterdorfer 54907ba352 Mute FullRollingRestartIT#testFullRollingRestart()
Relates #25420
2017-06-27 10:41:48 +02:00
Daniel Mitterdorfer ef9d099ffd Mute IndexShardTests#testRelocatedShardCanNotBeRevivedConcurrently 2017-06-27 10:25:40 +02:00
Jason Tedor f27aba34bf Mark shutdown non-master nodes test as awaits fix
This commit marks a failing test as awaits fix. The test is failing due
to a primary shard not knowing its own local checkpoint in the global
checkpoint tracker after recovery. If such a shard becomes primary after
promotion, and is then subsequently relocated, it can lead to a
violation of an assertion that when the primary context is transferred
the knowledge of all in-sync local checkpoints is consistent with the
global checkpoint on the relocation target.

Relates #25415
2017-06-26 22:48:04 -04:00
Jason Tedor dfd241e0a6 Remove default path settings
This commit removes the default path settings for data and logs. With
this change, we now ship the packages with these settings set in the
elasticsearch.yml configuration file rather than going through the
default.path.data and default.path.logs dance that we went through in
the past.

Relates #25408
2017-06-26 21:43:20 -04:00
Jason Tedor cca18a2c35 Make plugin loading stricter
Today we load plugins reflectively, looking for constructors that
conform to specific signatures. This commit tightens the reflective
operations here, not allowing plugins to have ambiguous constructors.

Relates #25405
2017-06-26 21:42:53 -04:00
Jason Tedor 5a9fc8aa2a Remove path.conf setting
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.

Relates #25392
2017-06-26 15:18:29 -04:00
Jason Tedor e9e7007a51 Remove LongTuple
This commit removes an abstraction that was introduced when introducing
the primary context. As this abstraction is used in exactly one place,
we simply make that abstraction local to its usage so that we do not
accumulate yet another general abstraction with exactly one usage.

Relates #25402
2017-06-26 14:46:06 -04:00
Jason Tedor 56d3a5e6d8 Fix primary context sealing test
This commit updates some assertions in the primary context sealing test
after the restriction on updating allocation IDs from master and
updating global checkpoint on replica while sealed were removed.
2017-06-26 14:17:33 -04:00
Jason Tedor c6a03bc549 Introduce primary context (#25122)
* Introduce primary context

The target of a primary relocation is not aware of the state of the
replication group. In particular, it is not tracking in-sync and
initializing shards and their checkpoints. This means that after the
target shard is started, its knowledge of the replication group could
differ from that of the relocation source. In particular, this differing
view can lead to it computing a global checkpoint that moves backwards
after it becomes aware of the state of the entire replication
group. This commit addresses this issue by transferring a primary
context during relocation handoff.

* Fix test

* Add assertion messages

* Javadocs

* Barrier between marking a shard in sync and relocating

* Fix misplaced call

* Paranoia

* Better latch countdown

* Catch any exception

* Fix comment

* Fix wait for cluster state relocation test

* Update knowledge via upate local checkpoint API

* toString

* Visibility

* Refactor permit

* Push down

* Imports

* Docs

* Fix compilation

* Remove assertion

* Fix compilation

* Remove context wrapper

* Move PrimaryContext to new package

* Piping for cluster state version

This commit adds piping for the cluster state version to the global
checkpoint tracker. We do not use it yet.

* Remove unused import

* Implement versioning in tracker

* Fix test

* Unneeded public

* Imports

* Promote on our own

* Add tests

* Import

* Newline

* Update comment

* Serialization

* Assertion message

* Update stale comment

* Remove newline

* Less verbose

* Remove redundant assertion

* Tracking -> in-sync

* Assertions

* Just say no

Friends do not let friends block the cluster state update thread on
network operations.

* Extra newline

* Add allocation ID to assertion

* Rename method

* Another rename

* Introduce sealing

* Sealing tests

* One more assertion

* Fix imports

* Safer sealing

* Remove check

* Remove another sealed check
2017-06-26 14:09:15 -04:00
Igor Motov 2a4fb950df Tests: Fix array out of bounds exception in TemplateUpgradeServiceIT 2017-06-26 09:14:05 -04:00
Martijn van Groningen a34f5fa812
Move more token filters to analysis-common module
The following token filters were moved: stemmer, stemmer_override, kstem, dictionary_decompounder, hyphenation_decompounder, reverse, elision and truncate.

Relates to #23658
2017-06-26 09:02:16 +02:00
Ryan Ernst 1583f81047 Test: Allow merging mock secure settings (#25387)
While real secure settings (ie an ES keystore) cannot be merged
together, mocked secure settings can and need to be sometimes merged.
This commit adds a merge method to allow tests to merge together
multiple instances of secure settings.
2017-06-25 10:19:51 -07:00
Simon Willnauer 4e4a104f4a Remove remaining `index.mapper.single_type` setting usage from tests (#25388)
This change removes the remaining explicitly specified `index.mapper.single_type`
settings from tests in order to allow the removal of the setting.
This is the already approved part of #25375 broken out to simplfiy reviews on
2017-06-25 12:25:41 +02:00
Jason Tedor 43c190339a Remove dead logger prefix code
When Log4j 2 was introduced, we removed support for the system property
es.logger.prefix. Yet, some code was left behind. This commit removes
that dead code.

Relates #25377
2017-06-24 08:16:59 -04:00
Igor Motov 79a8336559 Tests: Improve stability and logging of TemplateUpgradeServiceIT tests (#25386)
Relates to #25382
2017-06-23 17:31:21 -04:00
markharwood 973530f953 Added unit test coverage for SignificantTerms (#24904)
Added unit test coverage for GlobalOrdinalsSignificantTermsAggregator, GlobalOrdinalsSignificantTermsAggregator.WithHash, SignificantLongTermsAggregator and SignificantStringTermsAggregator.
Removed integration test.

Relates #22278
2017-06-23 15:34:38 +01:00
Boaz Leskes 9ff1698aa7 testCreateShrinkIndex: removed left over debugging log line that violated linting 2017-06-23 12:14:39 +02:00
Boaz Leskes 0ebc49e8c6 testCreateShrinkIndex should make sure to use the right source stats when testing shrunk target 2017-06-23 11:05:59 +02:00
Tanguy Leroux 6a792d6d82 [Test] Add unit test for XContentParserUtilsTests.parseStoredFieldsValue (#25288) 2017-06-23 10:54:26 +02:00
Simon Willnauer 4ae426a552 Remove remaining `index.mapping.single_type=false` (#25369)
This change cleans up remaining tests  to not use index.mapping.single_type=false
but instead where applicable use a single type or markt the index as created
with a pre 6.x version.

Yet, there is still on leftover in the client tests that needs special attention.
See `org.elasticsearch.client.SearchIT`

Relates to #24961
2017-06-23 10:26:06 +02:00
Martijn van Groningen 9c511bc447
test: Replace OldIndexBackwardsCompatibilityIT#testOldClusterStates with a full cluster restart qa test
OldIndexBackwardsCompatibilityIT#testOldClusterStates tested whether global and index metadata could be read from data directory,
 this can also be tested in full cluster qa test that checks cluster state via api.

 Relates to #24939
2017-06-23 09:54:05 +02:00
Boaz Leskes d20cd6afcb ESIndexLevelReplicationTestCase.ReplicationAction#execute should send exceptions to it's listener rather than bubble them up
This is how TRA works as well.
2017-06-22 23:37:08 +02:00
Boaz Leskes fb8c767737 testRecoveryAfterPrimaryPromotion shouldn't flush the replica with extra operations
We don't yet have lucene rollbacks, so we can't bake those in
2017-06-22 23:24:43 +02:00
Simon Willnauer 59b625121b Ensure `InternalEngineTests.testConcurrentWritesAndCommits` doesn't pile up commits (#25367)
`InternalEngineTests.testConcurrentWritesAndCommits` can be very heavy on disks
if threads are slow and the main thread keeps on pulling commit points holding on
to many many segments. This commit adds some quadratic backoff to not pile up too many
commits and to make sure indexing threads can make progress. This also now doesn't do
busy waiting but waits on a latch with a timeout.

Closes #25110
2017-06-22 21:50:11 +02:00
Simon Willnauer a077fa9b07 [TEST] Add debug logging if an unexpected exception is thrown 2017-06-22 21:19:39 +02:00
Igor Motov e6e5ae6202 TemplateUpgraders should be called during rolling restart (#25263)
In #24379 we added ability to upgrade templates on full cluster startup. This PR invokes the same update procedure also when a new node first joins the cluster allowing to update templates on a rolling cluster restart as well.

Closes #24680
2017-06-22 14:55:28 -04:00
Jason Tedor 8dcb1f5c7c Initialize max unsafe auto ID timestamp on shrink
When shrinking an index we initialize its max unsafe auto ID timestamp
to the maximum of the max unsafe auto ID timestamps on the source
shards.

Relates #25356
2017-06-22 11:14:25 -04:00
Boaz Leskes d963882053 Enable a long translog retention policy by default (#25294)
#25147  added the translog deletion policy but didn't enable it by default. This PR enables a default retention of 512MB (same maximum size of the current translog) and an age of 12 hours (i.e., after 12 hours all translog files will be deleted). This increases to chance to have an ops based recovery, even if the primary flushed or the replica was offline for a few hours.

In order to see which parts of the translog are committed into lucene the translog stats are extended to include information about uncommitted operations.

Views now include all translog ops and guarantee, as before, that those will not go away. Snapshotting a view allows to filter out generations that are not relevant based on a specific sequence number.

Relates to #10708
2017-06-22 17:08:14 +02:00
Simon Willnauer 29e80eea40 Remove `index.mapping.single_type=false` from core/tests (#25331)
This change cleans up core tests to not use `index.mapping.single_type=false`
but instead where applicable use a single type or markt the index as created
with a pre 6.x version.

Relates to #24961
2017-06-22 16:48:16 +02:00
Jason Tedor 97a2c4523d Get short path name for native controllers
Due to limitations with CreateProcessW on Windows (ultimately used by
ProcessBuilder) with respect to maximum path lengths, we need to get the
short path name for any native controllers before trying to start them
in case the absolute path exceeds the maximum path length. This commit
uses JNA to invoke the necessary Windows API for this to start the
native controller using the short path.

To be precise about the limitation here, the MSDN docs for
CreateProcessW say for the command line parameter:

>The command line to be executed. The maximum length of this string is
>32,768 characters, including the Unicode terminating null character. If
>lpApplicationName is NULL, the module name portionof lpCommandLine is
>limited to MAX_PATH characters.

This is exactly how the Windows implementation of Process in the JDK
invokes CreateProcessW: with the executable name (lpApplicationName) set
to NULL.

Relates #25344
2017-06-22 07:59:58 -04:00
Yannick Welsch e41eae9f05 Live primary-replica resync (no rollback) (#24841)
Adds a replication task that streams all operations from the primary's global checkpoint to all replicas.
2017-06-22 13:35:34 +02:00
Adrien Grand 44e9c0b947 Upgrade to lucene-7.0.0-snapshot-ad2cb77. (#25349)
Most notable changes:
 - better update concurrency: LUCENE-7868
 - TopDocs.totalHits is now a long: LUCENE-7872
 - QueryBuilder does not remove the boolean query around multi-term synonyms:
   LUCENE-7878
 - removal of Fields: LUCENE-7500

For the `TopDocs.totalHits` change, this PR relies on the fact that the encoding
of vInts and vLongs are compatible: you can write and read with any of them as
long as the value can be represented by a positive int.
2017-06-22 12:35:33 +02:00
Jason Tedor cc67d027de Initialize sequence numbers on a shrunken index
Bringing together shards in a shrunken index means that we need to
address the start of history for the shrunken index. The problem here is
that sequence numbers before the maximum of the maximum sequence numbers
on the source shards can collide in the target shards in the shrunken
index. To address this, we set the maximum sequence number and the local
checkpoint on the target shards to this maximum of the maximum sequence
numbers. This enables correct document-level semantics for documents
indexed before the shrink, and history on the shrunken index will
effectively start from here.

Relates #25321
2017-06-21 13:40:45 -04:00
Nik Everett 4bbb7e828b Port most snapshot/restore static bwc tests to qa:full-cluster-restart (#25296)
Ports all of RepositoryUpgradabilityIT to qa:full-cluster-restart and ports as much of RestoreBackwardsCompatIT as possible into qa:full-cluster-restart.
2017-06-21 13:26:03 -04:00
Nik Everett bec1a49a54 Javadoc: ThreadPool doesn't reject while shutdown (#23678)
It caught me offguard yesterday that our executors won't always
reject when the ThreadPool is shutdown.
2017-06-21 12:21:48 -04:00
Tanguy Leroux 49ebd65548 Add backward compatibility indices for 5.4.2 2017-06-21 10:42:26 +02:00
Tanguy Leroux 8274cd67ab Add version v5.4.2 after release 2017-06-21 10:23:32 +02:00
Alexander Reelsen 68423989da IndexMetaData: Add internal format index setting (#25292)
This setting is supposed to ease index upgrades as it allows you
to check for a new setting called `index.internal.version` which
can be used to check before upgrading indices.
2017-06-21 09:30:46 +02:00
Simon Willnauer 86a544de3b Ensure we never read from a closed MockSecureSettings object (#25322)
If secure settings are closed after the node has been constructed
no key-store access is permitted. We should also try to be as close as possible
to the real behavior if we mock secure settings. This change also adds
the same behavior as bootstrap has to InternalTestCluster to ensure we fail
if we try to read from secure settings after the node has been constructed.
2017-06-21 08:14:38 +02:00
Simon Willnauer 406a15e7a9 Fix settings serialization to not serialize secure settings or not take the total size into account (#25323) 2017-06-21 08:13:56 +02:00
Jason Tedor 1f14d042f6 Initialize primary term for shrunk indices
Today when an index is shrunk, the primary terms for its shards start
from one. Yet, this is a problem as the index will already contain
assigned sequence numbers across primary terms. To ensure document-level
sequence number semantics, the primary terms of the target shards must
start from the maximum of all the shards in the source index. This
commit causes this to be the case.

Relates #25307
2017-06-20 15:12:39 -04:00
Guillaume Le Floch 93e29d290f Tests: Refactor NodeTests settings (#25309)
This pull request aims to use the method baseSettings already present in the class.
2017-06-20 15:17:52 +02:00
Jun Ohtani 62d1969595 Parse synonyms with the same analysis chain (#8049)
* [Analysis] Parse synonyms with the same analysis chain

Synonym Token Filter / Synonym Graph Filter tokenize synonyms with whatever tokenizer and token filters appear before it in the chain.

Close #7199
2017-06-20 21:50:33 +09:00
Nik Everett 3261586cac Tweak reindex cancel logic and add many debug logs (#25256)
I'm still trying to hunt down rare failures in the cancelation tests
for reindex and friends. Here is the latest:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+multijob-unix-compatibility/os=ubuntu/876/console

It doesn't show much, other than that one of the tasks didn't kill
itself when asked to cancel.

So I'm going a bit crazy with debug logging so that the next time this
comes up I can trace exactly what happened.

Additionally, this tweaks the logic around how rethrottles were
performed around cancel. Previously we set the `requestsPerSecond`
to `0` when we cancelled the task. That was the "old way" to set them
to inifity which was the intent. This switches that from `0` to
`Float.MAX_VALUE` which is the "new way" to set the `requestsPerSecond`
to infinity. I don't know that this is much better, but it feels better.
2017-06-19 18:46:42 -04:00
Jay Modi 0d6c47fe14 Keystore CLI should use the AddFileKeyStoreCommand for files (#25298)
This commit fixes a typo in the KeyStoreCli class. The add-file command was incorrectly set to use
the AddStringKeyStoreCommand instead of the AddFileKeyStoreCommand.
2017-06-19 12:43:26 -06:00
Yannick Welsch 1a20760d79 Simplify IndexShard indexing and deletion methods (#25249)
Indexing or deleting documents through the IndexShard interface is quite complex and error-prone. It requires multiple calls, e.g. first prepareIndexOnPrimary, then do some checks if mapping updates have occurred, then do the actual indexing using index(...) etc. Currently each consumer of the interface (local recovery, peer recovery, replication) has additional custom checks built around it to deal with mapping updates, some of which are even inconsistent. This commit aims at reducing the complexity by exposing a simpler interface on IndexShard. There are no more prepare*** methods and the mapping complexity is also hidden, but still giving callers a possibility to implement custom logic to deal with mapping updates.
2017-06-19 20:11:54 +02:00
David Kyle d1be2ecfdb Initialise empty lists in BaseTaskResponse constructor (#25290)
* Initialise empty lists in BaseTaskResponse constructor

* Remove little used default constructor which leaves uninitialised members
2017-06-19 16:37:21 +01:00
Luca Cavanna d9ec2a23c5 Remove (deprecated) support for '+' in index expressions (#25274)
Relates to #24515
2017-06-19 15:19:17 +02:00
Tanguy Leroux e4f4886d40 [Test] Extend parsing checks for DocWriteResponses (#25257)
This commit changes the parsing logic of DocWriteResponse, ReplicationResponse
and GetResult so that it skips any unknown additional fields (for forward compatibility 
reasons). This affects the IndexResponse, UpdateResponse,DeleteResponse and 
GetResponse objects.
2017-06-19 13:19:09 +02:00
Martijn van Groningen bcaa413b0b
test: Port the remaining old indices search tests to full cluster restart qa module
Also tweaked the qa module's gradle file to actually run bwc tests against all index compat versions.

Relates to #24939
2017-06-19 12:27:24 +02:00
Simon Willnauer dc02b32650 Simplify connection closing and cleanups in TcpTransport (#25250)
Today we maintain a map of open connections in order to close them when
a low level channel gets closed or handles a failure. We also spawn a thread due to some
tricky concurrency issues especially with respect to netty since they listener might
be called on a transport / boss thread. Executions on those threads must not be blocking
since otherwise we will likely deadlock the event processing which adds to the
complexity of the concurrency model in this class.

This change associates the connection with the close callback that every channel invokes
once it's closed which allows us to remove the connections map. A relaxed non-blocking
concurrency model in the connection close listener allows cleaning up connected nodes without
blocking on any lock.
2017-06-19 09:19:45 +02:00
Boaz Leskes 7291aba8ae enable debug logging for testMasterFailoverDuringIndexingWithMappingChanges 2017-06-18 22:40:13 +02:00
Jason Tedor 4c28e781dd Fix failing delete index test
This test is failing because delete /{index} requests no longer support
index matching an alias. This commit removes testing such requests again
aliases.

Closes #25284
2017-06-18 15:32:43 -04:00
Christoph Büscher 3f9f713b44 Add AwaitsFix on IndicesRequestIT due to #25284 2017-06-18 18:56:41 +02:00
Christoph Büscher e99ced06cc [Tests] Check that parsing aggregations works in a forward compatible way (#25219)
This change adds tests for the aggregation parsing that try to simulate that we
can parse existing aggregations in a forward compatible way in the future,
ignoring potential newly added fields or substructures to the xContent response.
2017-06-17 13:06:31 +02:00
Ali Beyad 0c697348f4 Adds AwaitsFix on snapshot test failing due to #25281 2017-06-16 16:57:01 -04:00
Simon Willnauer f18b0d293c Move TransportStats accounting into TcpTransport (#25251)
Today TcpTransport is the de-facto base-class for transport implementations.
The need for all the callbacks we have in TransportServiceAdaptor are not necessary
anymore since we can simply have the logic inside the base class itself. This change
moves the stats metrics directly into TcpTransport removing the need for low level
bytes send / received callbacks.
2017-06-16 22:34:11 +02:00
Nik Everett ecc87f613f Move pre-configured "keyword" tokenizer to the analysis-common module (#24863)
Moves the keyword tokenizer to the analysis-common module. The keyword tokenizer is special because it is used by CustomNormalizerProvider so I pulled it out into its own PR. To get the move to work I've reworked the lookup from static to one using the AnalysisRegistry. This seems safe enough.

Part of #23658.
2017-06-16 11:48:15 -04:00
Luca Cavanna b5cea6980b Delete index API to work only against concrete indices (#25268)
With #23997 we have introduced a new internal index option that allows to resolve index expressions only against concrete indices while ignoring aliases. Such index option was applied to IndicesAliasesRequest, so that the index part of alias actions would only be resolved against concrete indices.

Same is done in this commit with delete index request. Deleting aliases has always been confusing as some users expect it to only remove the alias from the index (which has its own specific API). Even worse, in case of filtered aliases, deleting an alias may leave users with the expectation that only the documents that match the filter are deleted, which was never the case. To address all this confusion, delete index api works now only against concrete indices. WIldcard expressions will be only resolved against concrete index, as if aliases didn't exist. If one tries to delete against an alias, an IndexNotFoundException will be thrown regardless of whether the alias exists or not, as a concrete index with such a name doesn't exist.

Closes #2318
2017-06-16 17:46:01 +02:00
Boaz Leskes 9ddea539f5 Introduce translog size and age based retention policies (#25147)
This PR extends the TranslogDeletionPolicy to allow keeping the translog files longer than what is needed for recovery from lucene. Specifically, we allow specifying the total size of the files and their maximum age (i.e., keep up to 512MB but no longer than 12 hours). This will allow making ops based recoveries more common. 

Note that the default size and age still set to 0, maintaining current behavior. This is needed as the other components in the system are not yet ready for a longer translog retention. I will adapt those in follow up PRs.

Relates to #10708
2017-06-16 09:09:51 +02:00
Ali Beyad 350125ed2a Improves snapshot logging and snapshoth deletion error handling (#25264)
This commit does two things:
  1. Adds logging at the DEBUG level for when the index-N blob is
  updated.
  2. When attempting to delete a snapshot, if the snapshot was not found
  in the repository data, an exception is now thrown instead of silently
  ignoring the lack of presence of the snapshot in the repository data.
2017-06-15 19:43:19 -04:00
Christoph Büscher d3442f7d0c Add unit test for PathHierarchyTokenizerFactory (#24984) 2017-06-15 19:18:33 +02:00
Guillaume Le Floch a9014dfcc5 Deprecate tribe service
This commit deprecates the tribe service so that deprecation log
messages are delivered if a tribe node is configured.

Relates #24598
2017-06-15 12:41:05 -04:00
Martijn van Groningen 428e70758a
Moved more token filters to analysis-common module.
The following token filters were moved: `edge_ngram`, `ngram`, `uppercase`, `lowercase`, `length`, `flatten_graph` and `unique`.

Relates to #23658
2017-06-15 18:28:31 +02:00
Jim Ferenczi 2a78b0a19f [Test] Make sure that SearchAfterSortedDocQueryTests uses a single threaded searcher 2017-06-15 18:13:38 +02:00
markharwood 7a3155368c Test fix - removed superfluous assertion (#25247)
Closes #25245
2017-06-15 16:29:25 +01:00
Martijn van Groningen fe02829aac test: Ported more OldIndexBackwardsCompatibilityIT tests to full cluster restart qa tests. (#25173)
Relates to #24939
2017-06-15 14:48:06 +02:00
Adrien Grand 1b90c46a53 Allow reader wrappers to have different live docs but the same cache key.
Relates to #19856
2017-06-15 13:51:46 +02:00
Boaz Leskes 648b4717a4 move assertBusy to use CheckException (#25246)
We use assertBusy in many places where the underlying code throw exceptions. Currently we need to wrap those exceptions in a RuntimeException which is ugly.
2017-06-15 13:24:07 +02:00
Tanguy Leroux 27f1206999 Use SPI in High Level Rest Client to load XContent parsers (#25098)
This commit adds a NamedXContentProvider interface that can 
be implemented by plugins or modules using Java's SPI feature 
in order to provide additional NamedXContent parsers to external
applications like the Java High Level Rest Client.
2017-06-15 12:50:02 +02:00
Adrien Grand 5a6fa62844 Speed up PK lookups at index time. (#19856)
At index time Elasticsearch needs to look up the version associated with the
`_id` of the document that is being indexed, which is often the bottleneck for
indexing.

While reviewing the output of the `jfr` telemetry from a Rally benchmark, I saw
that significant time was spent in `ConcurrentHashMap#get` and `ThreadLocal#get`.
The reason is that we cache lookup objects per thread and segment, and for every
indexed document, we first need to look up the cache associated with this
segment (`ConcurrentHashMap#get`) and then get a state that is local to the
current thread (`ThreadLocal#get`). So if you are indexing N documents per
second and have S segments, both these methods will be called N*S times per
second.

This commit changes version lookup to use a cache per index reader rather than
per segment. While this makes cache entries live for less long, we now only need
to do one call to `ConcurrentHashMap#get` and `ThreadLocal#get` per indexed
document.
2017-06-15 10:17:42 +02:00
Adrien Grand 0c117145f6 Upgrade to lucene-7.0.0-snapshot-92b1783. (#25222)
This snapshot has faster range queries on range fields (LUCENE-7828), more
accurate norms (LUCENE-7730) and the ability to use fake term frequencies
(LUCENE-7854).
2017-06-15 09:52:07 +02:00
Ryan Ernst caf7792db1 Scripting: Rename SearchScript.needsScores to needs_score (#25235)
This commit renames the needsScores method so as to make it
automatically generatable, based on the name of the `_score` variable
which is available in search scripts. It also adds documentation to
ScriptContext to explain the naming and signature of such methods.
2017-06-14 22:01:19 -07:00
Jim Ferenczi 68deda6d03 FastVectorHighlighter should not cache the field query globally (#25197)
This commit removes the global caching of the field query and replaces it with
a caching per field. Each field can use a different `highlight_query` and the rewriting of
some queries (prefix, automaton, ...) depends on the targeted field so the query used for highlighting
must be unique per field.
There might be a small performance penalty when highlighting multiple fields since the query needs to be rewritten
once per highlighted field with this change.

Fixes #25171
2017-06-15 00:33:01 +02:00
Lee Hinman 4a30e23365 Remove QUERY_AND_FETCH BWC for pre-5.3.0 nodes (#25223)
* Remove QUERY_AND_FETCH BWC for pre-5.3.0 nodes

This was a BWC layer where we expicitly set the `search_type` to
"query_and_fetch" when a single node is queried on pre-5.3 nodes. Since 6.0 no
longer needs to be compatible with 5.3 nodes, this can be removed.

* Fix indentation

* Remove unused QUERY_FETCH_ACTION_NAME constant
2017-06-14 15:42:29 -06:00
Zachary Tong 52719b2118 Add more missing AggregationBuilder getters (#25198)
* Add more missing AggregationBuilder getters

- getMetadata for all aggs
- various getters on TermsAggBuilder (without "get" prefix to maintain convention)
- Also makes InternalSum's ctor public, to follow suit of other metrics (min/max/avg/etc)
2017-06-14 14:31:01 -04:00
Lee Hinman aa3134c093 Refactor TransportShardBulkAction.executeUpdateRequest and add tests
This splits `executeUpdateRequest` into separate parts and adds some unit tests
for the behavior in it. The actual behavior has not been changed.
2017-06-14 09:27:58 -06:00
Adrien Grand cadd31b3a8 Make sure range queries are correctly profiled. (#25108)
We introduced a new API for ranges in order to be able to decide whether points
or doc values would be more appropriate to execute a query, but since
`ProfileWeight` does not implement this API, the optimization is disabled when
profiling is enabled.
2017-06-14 16:31:16 +02:00
Martijn van Groningen e333955557
Remove PrefixAnalyzer, because it is no longer used. 2017-06-14 08:59:10 +02:00
Ryan Ernst 9ec1fc7b02 Internal: Remove Strings.cleanPath (#25209)
This commit removes the cleanPath method, in favor of using java's
Path.normalize().
2017-06-13 21:09:45 -07:00
Simon Willnauer bc7ec68e76 Add Cross Cluster Search support for scroll searches (#25094)
To complete the cross cluster search capabilities for all search types and
function this change adds cross cluster search support for scroll searches.
2017-06-13 17:22:49 +02:00
Sergey Galkin 1c95cbc4e8 Rollover max docs should only count primaries (#24977)
max_doc condition for index rollover should use document count only from primary shards 

Fixes #24217
2017-06-13 14:30:46 +02:00
Simon Willnauer 01d7c217f6 Add remote cluster infrastructure to fetch discovery nodes. (#25123)
In order to add scroll support for cross cluster search we need
to resolve the nodes encoded in the scroll ID to send requests to the
corresponding nodes. This change adds the low level connection infrastructure
that also ensures that connections are re-established if the cluster is
disconnected due to a network failure or restarts.

Relates to #25094
2017-06-13 14:23:56 +02:00
Simon Willnauer 186c16ea41 Ensure pending transport handlers are invoked for all channel failures (#25150)
Today if a channel gets closed due to a disconnect we notify the response
handler that the connection is closed and the node is disconnected. Unfortunately
this is not a complete solution since it only works for published connections.
Connections that are unpublished ie. for discovery can indefinitely hang since we
never invoke their handers when we get a failure while a user is waiting for
the response. This change adds connection tracking to TcpTransport that ensures
we are notifying the corresponding connection if there is a failure on a channel.
2017-06-13 09:37:05 +02:00
Lee Hinman ee1113c902 Tweak AggregatorBase.addRequestCircuitBreakerBytes
This modifies a method Mark added to the AggregatorBase that allows aggregations
to add additional memory tracking for datastructures used during execution. If
an aggregation would like to reclaim circuit breaker reserved bytes by adding a
negative number, `addWithoutBreaking` should be used instead of
`addEstimateBytesAndMaybeBreak`.

Resolves #24511
2017-06-12 12:55:50 -06:00
Jason Tedor bb66f3b76b Explicitly reject duplicate data paths
Duplicate data paths already fail to work because we would attempt to
take out a node lock on the directory a second time which will fail
after the first lock attempt succeeds. However, how this failure
manifests is not apparent at all and is quite difficult to
debug. Instead, we should explicitly reject duplicate data paths to make
the failure cause more obvious.

Relates #25178
2017-06-12 12:55:19 -04:00
Jason Tedor 982900eabf Do not swallow node lock failed exception
When attempting to obtain the node lock, if an exception is thrown it is
not logged. This makes debugging difficult. This commit causes such an
exception to be logged.

Relates #25176
2017-06-12 11:42:45 -04:00
markharwood 518cda6637 Aggregations bug: Significant_text fails on arrays of text. (#25030)
* Aggregations bug: Significant_text fails on arrays of text.
The set of previously-seen tokens in a doc was allocated per-JSON-field string value rather than once per JSON document meaning the number of docs containing a term could be over-counted leading to exceptions from the checks in significance heuristics. Added unit test for this scenario

Closes #25029
2017-06-12 14:02:54 +01:00
Jim Ferenczi 7ab3d5d04a Speed up sorted scroll when the index sort matches the search sort (#25138)
Sorted scroll search can use early termination when the index sort matches the scroll search sort.
The optimization can be done after the first query (which still needs to collect all documents)
by applying a query that only matches documents that are greater than the last doc retrieved in the previous request.
Since the index is sorted, retrieving the list of documents that are greater than the last doc
only requires a binary search on each segment.
This change introduces this new query called `SortedSearchAfterDocQuery` and apply it when possible.
Scrolls with this optimization will search all documents on the first request and then will early terminate each segment
after $size doc for any subsequent requests.

Relates #6720
2017-06-12 09:33:30 +02:00
Boaz Leskes f34136eda4 TranslogTests.testWithRandomException ignored a possible simulated OOM when trimming files 2017-06-12 08:32:55 +02:00
Boaz Leskes cfb5f6a5a6 Adapt TranslogTests.testWithRandomException to checkpoint syncing on trim
#25005 changed the translog dynamic to fsync the checkpoint before trimming a file. This changed the dynamics of potential failure modes which requires a change to testWithRandomException - it's now possible that we had an exception but the translog was trimmed.

Closes #25133
2017-06-11 23:17:10 +02:00
Jason Tedor dcf57f296e Fix get mappings HEAD requests
Get mappings HEAD requests incorrectly return a content-length header of
0. This commit addresses this by removing the special handling for get
mappings HEAD requests, and just relying on the general mechanism that
exists for handling HEAD requests in the REST layer.

Relates #23192
2017-06-11 14:58:56 -04:00
Boaz Leskes 9b8754e4c2 TranslogTests#commit didn't allow for a concurrent closing of a view
The view closing will trim unneeded files but there is a small window where they may still be around.
2017-06-11 19:09:01 +02:00
Jason Tedor 7182577904 Fix handling of exceptions thrown on HEAD requests
Today when an exception is thrown handling a HEAD request, the body is
swallowed before the channel has a chance to see it. Yet, the channel is
where we compute the content length that would be returned as a header
in the response. This is a violation of the HTTP specification. This
commit addresses the issue. To address this issue, we remove the special
handling in bytes rest response for HEAD requests when an exception is
thrown. Instead, we let the upstream channel handle the special case, as
we already do today for the non-exceptional case.

Relates #25172
2017-06-10 23:44:18 -04:00
Jason Tedor 5108fa7529 Remove unneeded weak reference from prefix logger
We have a custom logger implementation known as a prefix logger that is
used to write every message by the logger with a given prefix. This is
useful for node-level, index-level, and shard-level messages where we
want to log the node name, index name, and shard ID, respectively, if
possible. The mechanism that we employ is that of a marker. Log4j has a
built-in facility for managing these markers, but its effectively a
memory leak because these markers are held in a map and can never be
released. This is problematic for us since indices and shards do not
necessarily have infinite life spans and so on a node where there are
many indices being creted and destroyed, this infinite lifespan can be a
problem indeed. To solve this, we use our own cache of markers. This is
necessary to prevent too many instances of the marker for the same
prefix from being created (just think of all the shard-level components
that exist in the system), and to workaround the effective leak in
Log4j. These markers are stored as weak references in a weak hash
map. It is these weak references that are unneeded. When a key is
removed from a weak hash map, the corresponding entry is placed on a
reference queue that is eventually cleared. This commit simplifies
prefix logger by removing this unnecessary weak reference wrapper.

Relates #22460
2017-06-10 13:20:45 -04:00
Chris Earle af7b479e12 "shard started" should show index and shard ID (#25157)
When the cluster state is updated with Shard Started entries, it simply adds "shard-started" as the source of the change.

This adds the index name and shard ID so that we can see who/what is spamming the changes when the index creation step has already left the cluster state.
2017-06-09 14:52:42 -04:00
Boaz Leskes b8fef3309c await fix testWithRandomException 2017-06-09 20:31:39 +02:00
Jason Tedor 8a45c3105f Change BWC versions on create index response
This commit changes the BWC versions on the create index response now
that the index name in the response is supported since 5.6.0.

Relates #25139
2017-06-09 13:52:08 -04:00
Sergey Novikov 7c8657df0e Return the index name on a create index response
This commit modifies the create index response so that it includes the
index name.

Relates #25139
2017-06-09 13:47:47 -04:00
Koen De Groote 64888f6f01 Correctly format arrays in output
There are a few places where arrays are output in messages yet the
output would merely use the default toString implementation rather than
actually putting the content of the array in the message. This commit
fixes the issue.

Relates #24340
2017-06-09 11:45:07 -04:00
Christoph Büscher 823cbb437b [Test] Extending parsing checks for SearchResponse (#25148)
This change extends the tests and parsing of SearchResponse to make sure we can
skip additional fields the parser doesn't know for forward compatibility
reasons.
2017-06-09 17:33:44 +02:00
Ryan Ernst a03b6c2fa5 Scripting: Change keys for inline/stored scripts to source/id (#25127)
This commit adds back "id" as the key within a script to specify a
stored script (which with file scripts now gone is no longer ambiguous).
It also adds "source" as a replacement for "code". This is in an attempt
to normalize how scripts are specified across both put stored scripts and script usages, including search template requests. This also deprecates the old inline/stored keys.
2017-06-09 08:29:25 -07:00
Martijn van Groningen c7ae27d57f
nested: In case of a single type the _id field should be added to the nested document instead of _uid field.
When `index.mapping.single_type` is `true` the `_uid` field is not used and instead `_id` field is used.
Prior to this change nested documents would in this case still use the `_uid` field to mark to what root
document they belong to. In case of deleting documents this could lead to only the root  Lucene document
to be deleted and not the nested Lucene documents. This broke the docid block ordering the block join
relies on in order to work correctly and thus causing the `nested` query, `nested` aggregation, nested sorting
and nested inner hits to either fail or yield incorrect results.

This bug only manifests in 6.0.0-ALPHA2 release and snaphots (5.5.0-SNAPSHOT, 5.6.0-SNAPSHOT, 6.0.0-SNAPSHOT).
2017-06-09 14:57:11 +02:00
Adrien Grand 87d19b21c7 `type` and `id` are lost upon serialization of `Translog.Delete`. (#24586)
This was introduced in #24460: the constructor of `Translog.Delete` that takes
a `StreamInput` does not set the type and id. To make it a bit more robust, I
made fields final so that forgetting to set them would make the compiler
complain.
2017-06-09 14:56:23 +02:00
Sergey Galkin dc5aa993e0 Fix NPE in token_count datatype with null value (#25046)
Fixes an issue with the handling of null values for the token_count data type.

Closes #24928
2017-06-09 14:13:05 +02:00
Jim Ferenczi 8250aa4267 Remove the postings highlighter and make unified the default highlighter choice (#25028)
This change removes the `postings` highlighter. This highlighter has been removed from Lucene master (7.x) because it behaves
exactly like the `unified` highlighter when index_options is set to `offsets`:
https://issues.apache.org/jira/browse/LUCENE-7815

It also makes the `unified` highlighter the default choice for highlighting a field (if `type` is not provided).
The strategy used internally by this highlighter remain the same as before, it checks `term_vectors` first, then `postings` and ultimately it re-analyzes the text.
Ultimately it rewrites the docs so that the options that the `unified` highlighter cannot handle are clearly marked as such.
There are few features that the `unified` highlighter is not able to handle which is why the other highlighters (`plain` and `fvh`) are still available.
I'll open separate issues for these features and we'll deprecate the `fvh` and `plain` highlighters when full support for these features have been added to the `unified`.
2017-06-09 14:09:57 +02:00
Christoph Büscher eca4f24b16 [Test] Adding test for parsing SearchShardFailure leniently (#25144)
This change extends the tests and parsing of SearchShardFailure to make sure we
can skip fields the parser doesn't know for forward compatibility reasons.
2017-06-09 12:46:09 +02:00
Christoph Büscher 79057b1c61 [Test] Extending checks for Suggestion parsing (#25132)
When parsing responses we should be ignoring any new unknown fields or inner
objects in most cases to be forward compatible with changes in core on the
client side. This change adds test for this for Suggestions and its various
subclasses to check if we are able to ignore new fields and objects in the
xContent.
2017-06-09 10:11:08 +02:00
Tal Levy 340909582f remove Ingest's Internal Template Service (#25085)
Ingest was using it's own wrapper around TemplateScripts and the ScriptService.
This commit removes that abstraction
2017-06-08 15:24:03 -07:00
Lee Hinman 119f8ed9f0 Correctly enable _all for older 5.x indices
When we disabled `_all` by default for indices created in 6.0, we missed adding
a layer that would handle the situation where `_all` was not enabled in 5.x and
then the cluster was updated to 6.0, this means that when the cluster was
updated the `_all` field would be disabled for 5.x indices and field values
would not be added to the `_all` field.

This adds a compatibility layer for 5.x indices where we treat the default
enabled value for the `_all` field to be `true` if unset on 5.x indices.

Resolves #25068
2017-06-08 14:37:44 -06:00
Jason Tedor 1708f1773b Mark Log4j API dependency as non-optional
The Log4j dependency is separated into two artifacts, the API and the
core implementation. This is to enable replacing Log4j on the backend
through the SLF4J bridge with another logging implementation. For this
reason, the dependencies are marked as optional. This causes confusion
amongst users as to use the bridge, the API should be non-optional since
it is needed for the bridge to function correctly. While they could pull
it into their application directly, it would be clearer if we simply
marked this depdendency as non-optional. Note that this does not mean
that users have to use Log4j for logging in their application, so we are
not marking core as required, it only clarifies what they need to be
able to plug in a different logging implementation.

Relates #25136
2017-06-08 16:09:34 -04:00
Lee Hinman 050b7cd0f9 Include empty mappings in GET /{index}/_mappings requests (#25118)
Previously this would output:

```
GET /test-1/_mappings

{ }
```

And after this change:

```
GET /test-1/_mappings

{
  "test-1": {
    "mappings": {}
  }
}
```

To bring parity back to the REST output after #24723.

Relates to #25090
2017-06-08 10:57:04 -06:00
Lee Hinman 5b2ab96364 Return index name and empty map for /{index}/_alias with no aliases
Previously in #24723 we changed the `_alias` API to not go through the
`RestGetIndicesAction` endpoint, instead creating a `RestGetAliasesAction` that
did the same thing.

This changes the formatting so that it matches the old formatting of the
endpoint, before:

```
GET /test-1/_alias

{ }
```

And after this change:

```
GET /test-1/_alias

{
  "test-1": {
    "aliases": {}
  }
}
```

This is related to #25090
2017-06-08 10:03:03 -06:00
Eli Skeggs ee0e921643 Fix typo in GeoUtils#isValidLongitude (#25121)
GeoUtils#isValidLongitude is inconsistent with GeoUtils#isValidLatitude. 
Neither technically need the isInfinite() check because they then compare 
against min and max values.
2017-06-08 17:23:22 +02:00
Christoph Büscher a0afa917ac [Tests] Check QueryProfileShardResult parser robustness for new fields (#25130)
When parsing resonses we should be ignoring any new unknown fields or inner
objects in most cases to be forward compatible with changes in core on the
client side. This change adds test for this for QueryProfileShardResult and
nested substructures and changes the parsing code where necessary to be able to
ignore new fields and objects in the xContent.
2017-06-08 16:40:00 +02:00
Nik Everett 4a8c09c5f1 Make randomVersionBetween work with unreleased versions (#25042)
Test: randomVersionBetween works with unreleased

Modifies randomVersionBetween so that it works with unreleased
versions. This should make switching a version from unreleased
to released much simpler.
2017-06-08 10:19:06 -04:00
Yannick Welsch cd57395c98 Use correct primary term for replicating NOOPs (#25128)
NOOPs should be, same as for indexing operations, written on the replica using the original operation term instead of the current term of the replica.
2017-06-08 14:20:26 +02:00
Martijn van Groningen 326fa33d4e
fielddata: Binary script doc values should make a deep copy of the BytesRef before populating it in the values array.
Added common base class for ScriptDocValues.Strings and ScriptDocValues.BytesRefs now that these classes are very similar.

Also cleaned up the BinaryDVFieldDataTests:
* Use junit assertions instead of hamcrest
* Use BytesRef directly instead of byte[]

Closes #24785
2017-06-08 13:20:35 +02:00
Jim Ferenczi eeac4b9721 Fix Fast Vector Highlighter NPE on match phrase prefix (#25116)
The FVH fails with an NPE when a match phrase prefix is rewritten in an empty phrase query.
This change makes sure that the multi match query rewrites to a MatchNoDocsQuery (instead of an empty phrase query) when there is
a single term and that term does not expand to any term in the index.

Fixes #25088
2017-06-08 12:27:11 +02:00
Jim Ferenczi 36a5cf8f35 Automatically early terminate search query based on index sorting (#24864)
This commit refactors the query phase in order to be able
to automatically detect queries that can be early terminated.
If the index sort matches the query sort, the top docs collection is early terminated
on each segment and the computing of the total number of hits that match the query is delegated to a simple TotalHitCountCollector.
This change also adds a new parameter to the search request called `track_total_hits`.
It indicates if the total number of hits that match the query should be tracked.
If false, queries sorted by the index sort will not try to compute this information and 
and will limit the collection to the first N documents per segment.
Aggregations are not impacted and will continue to see every document
even when the index sort matches the query sort and `track_total_hits` is false.

Relates #6720
2017-06-08 12:10:46 +02:00
Jim Ferenczi 21a57c1494 Always use DisjunctionMaxQuery to build cross fields disjunction (#25115)
This commit modifies query_string, simple_query_string and multi_match queries to always use a DisjunctionMaxQuery when a disjunction over multiple fields is built. The tiebreaker is set to 1 in order to behave like the boolean query in terms of scoring.
The removal of the coord factor in Lucene 7 made this change mandatory to correctly handle minimum_should_match.

Closes #23966
2017-06-08 11:18:17 +02:00
Simon Willnauer d6d416cacc Break out clear scroll logic from TransportClearScrollAction (#25125)
This change extracts the main logic from `TransportClearScrollAction`
into a new class `ClearScrollController` and adds a corresponding unit test.

Relates to #25094
2017-06-08 11:13:08 +02:00
Simon Willnauer bdc3a16fa4 Fix naminig in GroupedActionListener
GroupedActionListener still had some members named from it's specialization
before it was factored out in a general purpose class.
2017-06-08 10:21:15 +02:00
Adrien Grand a8ea2f0df4 Leverage scorerSupplier when applicable. (#25109)
The `scorerSupplier` API allows to give a hint to queries in order to let them
know that they will be consumed in a random-access fashion. We should use this
for aggregations, function_score and matched queries.
2017-06-08 10:19:38 +02:00
Boaz Leskes 087f182481 Translog file recovery should not rely on lucene commits (#25005)
When we open a translog, we rely on the `translog.ckp` file to tell us what the maximum generation file should be and on the information stored in the last lucene commit to know the first file we need to recover. This requires coordination and is currently subject to a race condition: if a node dies after a lucene commit is made but before we remove the translog generations that were unneeded by it, the next time we open the translog we will ignore those files and never delete them (I have added tests for this).

This PR changes the approach to have the translog store both of those numbers in the `translog.ckp`. This means it's more self contained and easier to control. 

This change also decouples the translog recovery logic from the specific commit we're opening. This prepares the ground to fully utilize the deletion policy introduced in #24950 and store more translog data that's needed for Lucene, keep multiple lucene commits around and be free to recover from any of them.
2017-06-08 09:21:28 +02:00
Simon Willnauer ce24331d1f Add helper methods to TransportActionProxy to identify proxy actions and requests (#25124)
Downstream users of out network intercept infrastructure need this information which is
hidden due to member and class visibility.
2017-06-08 09:07:22 +02:00
Jack Conradson d187fa78fd Generate Painless Factory for Creating Script Instances (#25120) 2017-06-07 16:06:11 -07:00
Christoph Büscher 9e741cd13d Tests: Add ability to generate random new fields for xContent parsing test (#23437)
For the response parsing we want to be lenient when it comes to parsing
new xContent fields. In order to ensure this in our testing, this change
adds a utility method to XContentTestUtils that takes xContent bytes
representation as input and recursively a random field on each object
level.

Sometimes we also want to exclude a whole subtree from this treatment 
(e.g. skipping "_source"), other times an element (e.g. "fields", "highlight" 
in SearchHit) can have arbitraryly named objects. Those cases can be 
specified as exceptions.
2017-06-07 21:01:20 +02:00
Jim Ferenczi 68f1d4df5a bump the Lucene version for Version 5.5 and 5.6 after the upgrade to Lucene 6.6.0 2017-06-07 19:32:13 +02:00
Ryan Ernst 2057bbc6c5 Scripting: Remove unnecessary intermediate script compilation methods on QueryShardContext (#25093)
This commit removes wrapper methods on QueryShardContext used to compile
scripts. Instead, the script service is made accessible in the context,
and calls to compile can be made directly. This will ease transition to
each of those location becoming their own context, since they would no
longer be able to expect the same script class type.
2017-06-07 08:24:18 -07:00
Yannick Welsch 26ec89173b Remove TranslogRecoveryPerformer (#24858)
Splits TranslogRecoveryPerformer into three parts:
- the translog operation to engine operation converter
- the operation perfomer (that indexes the operation into the engine)
- the translog statistics (for which there is already RecoveryState.Translog)
This makes it possible for peer recovery to use the same IndexShard interface as bulk shard requests (i.e. Engine operations instead of Translog operations). It also pushes the "fail on bad mapping" logic outside of IndexShard. Future pull requests could unify the BulkShard and peer recovery path even more.
2017-06-07 17:11:27 +02:00
Jim Ferenczi c8bf7ecaed Higlighters: Fix MultiPhrasePrefixQuery rewriting (#25103)
The unified highlighter rewrites MultiPhrasePrefixQuery to SpanNearQuer even when there is a single term in the phrase.
Though SpanNearQuery throws an exception when the number of clauses is less than 2.
This change returns a simple PrefixQuery when there is a single term and builds the SpanNearQuery otherwise.

Relates #25088
2017-06-07 16:14:28 +02:00
Tim Brooks 233c63fc63 Add version 5.6 to versions (#25084)
* Add version 5.6 to versions

* Fix test

* Remove 5.4.2 constant
2017-06-07 09:59:27 -04:00
Boaz Leskes 8e15186293 Update `IndexShard#refreshMetric` via a `ReferenceManager.RefreshListener` (#25083)
The PR takes a different approach to solve #24806 than currently implemented via #25052. The `refreshMetric` that IndexShard maintains is updated using the refresh listeners infrastructure in lucene. This means that we truly count all refreshes that lucene makes and not have to worry about each individual caller (like `IndexShard@refresh` and `Engine#get()`)
2017-06-07 10:54:10 +02:00
Martijn van Groningen db8aa8e94e
Changed inner_hits to work with the new join field type and
at the same time maintaining support for the `_parent` meta field type/

Relates to #20257
2017-06-07 10:52:49 +02:00
Yu 14913fdc37 keep _parent field while updating child type mapping (#24407)
parent/child: Allow updating mapping without specifying `_parent` field on each update.

Prior to this change when a mapping has a `_parent` field then any update (also updates that didn't modify the `_parent` field) to the mapping involved specifying the `_parent` field again. With this change specifying the `_parent` field on each mapping update is no longer required.

Closes #23381
2017-06-07 10:51:21 +02:00
Jason Tedor 2f5f27fafa Remove unnecessary callback interface
We have a callback interface that is not needed because it is
effectively the same as java.util.function.Consumer. This commit removes
it.

Relates #25089
2017-06-06 20:50:03 -04:00
Tim Brooks feca0a9f33 Bumping version to v6.0.0-alpha3 (#25077) 2017-06-06 15:47:23 -05:00
Jason Tedor 1a681a928d Modify cluster state callback in recovery land
We use a callback in recovery land during primary relocation to ensure
the relocation target is on at least the same version as the relocation
source. This callback is typed as a Callback<Long> which is an
unnecessary custom type (we can use Consumer<T> or the appropriate
primitive callbacks). Here, we can use LongConsumer.

Relates #25081
2017-06-06 16:29:10 -04:00
Jason Tedor e03c4938c5 GET aliases should 404 if aliases are missing
Previously the HEAD and GET aliases endpoints were misaigned in
behavior. The HEAD verb would 404 if any aliases are missing while the
GET verb would not if any aliases existed. When HEAD was aligned with
GET, this broke the previous usage of HEAD to serve as an existence
check for aliases. It is the behavior of GET that is problematic here
though, if any alias is missing the request should 404. This commit
addresses this by modifying the behavior of GET to behave in this
way. This fixes the behavior for HEAD to also 404 when aliases are
missing.

Relates #25043
2017-06-06 14:37:29 -04:00
Jim Ferenczi 7e60cf3e54 Move parent_id query to the parent-join module (#25072)
This change moves the parent_id query to the parent-join module and handles the case when only the parent-join field can be declared on an index (index with single type on).
If single type is off it uses the legacy parent join field mapper and switch to the new one otherwise (default in 6).

Relates #20257
2017-06-06 19:35:14 +02:00
Ryan Ernst 7ec39acd4b Settings: Fix setting groups to include secure settings (#25076)
This commit fixes the group methdos of Settings to properly include
grouped secure settings. Previously the secure settings were included
but without the group prefix being removed.

closes #25069
2017-06-06 10:13:10 -07:00
Yu 40a13345d7 Add refresh stats tracking for realtime get (#25052)
Passes a `LongConsumer` into the `Engine` during GETs which the engine
calls if it refreshed to perform the get.

Closes #24806
2017-06-06 12:39:02 -04:00
olcbean 0d5f3958e7 Expand index expressions against indices only when managing aliases (#23997)
The index parameter in the update-aliases, put-alias, and delete-alias APIs no longer accepts alias names. Instead, it accepts only index names (or wildcards which will expand to matching indices).

Closes #23960
2017-06-06 11:01:38 +02:00
Ryan Ernst ac82824d80 Settings: Fix secure settings by prefix (#25064)
This commit fixes a bug in retrieving a sub Settings object for a given
prefix with secure settings. Before this commit the returned Settings
would be filtered by the prefix, but the found setting names would not
have the prefix removed.
2017-06-06 00:11:33 -07:00
Lee Hinman b6a2b8d682 Track EWMA[1] of task execution time in search threadpool executor
This is the first step towards adaptive replica selection (#24915). This PR
tracks the execution time, also known as the "service time" of a task in the
threadpool. The `QueueResizingEsThreadPoolExecutor` then stores a moving average
of these task times which can be retrieved from the executor.

Currently there is no functionality using the EWMA yet (other than tests), this
is only a bite-sized building block so that it's easier to review.

[1]: EWMA = Exponentially Weighted Moving Average
2017-06-05 10:09:41 -06:00
Ali Beyad f2a23e3459 Removes an invalid assert in resizing big arrays which does not always
hold (resizing can result in a smaller size than the current size, while
the assert attempted to verify the new size is always greater than the
current).
2017-06-05 11:49:06 -04:00
Alex Benusovich 5463294ec4 Fixed NPEs caused by requests without content. (#23497)
REST handlers that require a body will throw an an ElasticsearchParseException "request body required".
REST handlers that require a body OR source param will throw an ElasticsearchParseException "request body or source param required".
Replaced asserts in BulkRequest parsing code with a more descriptive IllegalArgumentException if the line contains an empty object.
Updated bulk REST test to verify an empty action line is rejected properly.
Updated BulkRequestTests with randomized testing for an empty action line.
Used try-with-resouces for XContentParser in AbstractBulkByQueryRestHandler.
2017-06-05 09:08:14 -06:00
Nik Everett 73307a2144 Plugins can register pre-configured char filters (#25000)
Fixes the plumbing so plugins can register char filters and moves
the `html_strip` char filter into analysis-common.

Relates to #23658
2017-06-05 09:25:15 -04:00
Ryan Ernst e22a68295c Tests: Make secure settings available from settings builder for tests (#25037)
This commit exposes the secure settings in Settings.Builder, so that
the current secure settings can be retrieved and added to when creating
settings for tests. This is necessary since secure settings can only be
added once to a builder, so chains of methods using settings builders
must reuse the already set mock secure settings.
2017-06-03 16:55:34 -07:00
Sergey Novikov 57b4002357 Include duplicate jar when jarhell check fails
When the jarhell check fails due to a duplicate jar on the classpath,
the exception message includes the full classpath but not the duplicated
jar. For a long classpath, this can make it difficult to find the jar
that is duplicated. This commit changes the exception message to include
the duplicated jar.

Relates #24953
2017-06-02 18:22:01 -04:00
Lee Hinman a32d1b91fa Remove comma-separated feature parsing for GetIndicesAction
This removes the parsing of things like `GET /idx/_aliases,_mappings`, instead,
a user must choose between retriving all index metadata with `GET /idx`, or only
a specific form such as `GET /idx/_settings`.

Relates to (and is a prerequisite of) #24437
2017-06-02 14:43:38 -06:00
Ryan Ernst 0d8216d5af Scripting: Convert CompiledTemplate to a ScriptContext (#25032)
This commit creates TemplateScript and associated classes so that
templates no longer need a special ScriptService.compileTemplate method.
The execute() method is equivalent to the old run() method.

relates #20426
2017-06-02 13:41:26 -07:00
Ali Beyad e024c67561 Checks the circuit breaker before allocating bytes for a new big array (#25010)
Previously, when allocating bytes for a BigArray, the array was created
(or attempted to be created) and only then would the array be checked
for the amount of RAM used to see if the circuit breaker should trip.

This is problematic because for very large arrays, if creating or
resizing the array, it is possible to attempt to create/resize and get
an OOM error before the circuit breaker trips, because the allocation
happens before checking with the circuit breaker.

This commit ensures that the circuit breaker is checked before all big
array allocations (note, this does not effect the array allocations that
are less than 16kb which use the [Type]ArrayWrapper classes found in
BigArrays.java).  If such an allocation or resizing would cause the
circuit breaker to trip, then the breaker trips before attempting to
allocate and potentially running into an OOM error from the JVM.

Closes #24790
2017-06-02 15:16:22 -04:00
Ali Beyad 3cb307462d Consolidates the logic for cleaning up snapshots on master election (#24894)
In #24605, logic was implemented to ensure that completed snapshots were
properly removed from the cluster state upon a change in master nodes.
This commit removes redundant logic that also attempted to clean up
completed snapshots from the cluster state on master election, but only
covered a limited case that was remedied in #24605.

This commit also adds a test to ensure cleaning up of completed
snapshots at the right moment in time when a master election happens
before finalizing a snapshot, as well as adds a check to handle the case
where the old master and new master could attempt to finalize the
snapshot and write the same blob to the repository simultaneously.
2017-06-02 14:51:13 -04:00
Chris Earle 6ea9d83b2d Remove @Override that doesn't exist in parent anymore from new TransportNodesUsageAction 2017-06-02 10:19:17 -04:00
Chris Earle 6464add551 Always Accumulate Transport Exceptions (#25017)
This removes the `accumulateExceptions()` method (and its usage) from `TransportNodesAction` and `TransportTasksAction`, forcing both transport actions to always accumulate exceptions.

Without this change, some transport actions, like `TransportNodesStatsAction` would respond in very unexpected ways by returning no response due to some failure, but instead of returning an
error the response would simply be empty: no response and no error.

This results in a very trappy response structure where users can check for an error, then attempt to blindly use the response when no error is returned.
2017-06-02 10:01:42 -04:00
Tanguy Leroux 5f3ed99c71 [Test] Reduce number of buckets in SearchResponseTests and AggregationsTests (#24964)
This commit reduces the number of buckets that are generated for multi
 bucket aggregations in AggregationsTests and SearchResponseTests.

 The number of buckets are now limited to a maximum of 3 but before some
 aggregations could generate up to 10 buckets.
2017-06-02 15:59:25 +02:00
Jim Ferenczi b8605775df Add the ability to set eager_global_ordinals in the new parent-join field (#25019)
Defaults to true
2017-06-02 15:34:22 +02:00
Jason Tedor 7ebba35c32 Handle already closed while filling gaps
We can hit an already closed exception when filling the gaps after
blocking operations when updating the primary term on a promoted replica
shard. We should catch this and suppress it as it is an expected outcome
instead of letting it bubble up which leads to trying to fail the shard
which throws yet another already closed exception.

Relates #25021
2017-06-02 08:05:33 -04:00
olcbean 6dea5f14c3 Java api: Remove unneeded getTookInMillis method (#23923)
Some response classes in the java api expose both `getTook()` which returns a `TimeValue` and `getTookInMillis` which returns a `long` value. `getTook()` is enough as one can do `getTook().millis()` to obtain the same result as `getTookInMillis()`, which can be removed.
2017-06-02 11:11:05 +02:00
Colin Goodheart-Smithe 779fb9a1c0 Adds nodes usage API to monitor usages of actions (#24169)
* Adds nodes usage API to monitor usages of actions

The nodes usage API has 2 main endpoints

/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.

At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.

Still to do:

* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation

* Rafactors UsageService so counts are done by the handlers

* Fixing up docs tests

* Adds a name to all rest actions

* Addresses review comments
2017-06-02 08:46:38 +01:00
Tanguy Leroux 528bd25fa7 Add superset size to Significant Term REST response (#24865)
This commit adds a new bg_count field to the REST response of
SignificantTerms aggregations. Similarly to the bg_count that already
exists in significant terms buckets, this new bg_count field is set at
the aggregation level and is populated with the superset size value.
2017-06-02 09:45:15 +02:00
Tanguy Leroux c66be4a951 [Test] Remove unused test resources in core (#25011)
It looks like many unnecessary files remain in the core test resources directory. This commit removes them.
2017-06-02 09:08:51 +02:00
Ryan Ernst 8d88b94372 Scripting: Add optional context parameter to put stored script requests (#25014)
This commit adds an optional `context` url parameter to the put stored
script request. When a context is specified, the script is compiled
against that context before storing, as a validation the script will
work when used in that context.
2017-06-01 17:53:48 -07:00
Simon Willnauer 39e59b49b1 Extract a common base class for scroll executions (#24979)
Today there is a lot of code duplication and different handling of errors
in the two different scroll modes. Yet, it's not clear if we keep both of
them but this simplification will help to further refactor this code to also
add cross cluster search capabilities.

This refactoring also fixes bugs when shards failed due to the node dropped out of the cluster in between scroll requests and failures during the fetch phase of the scroll. Both places where simply ignoring the failure and logging to debug. This can cause issues like #16555
2017-06-01 22:23:41 +02:00
Nik Everett 4fcead9a65 Add backwards compatibility indices
Adds backwards compatiblity indices and repos for the 5.4.1
and 5.3.3 release.
2017-06-01 12:34:03 -04:00
Jason Tedor 0435ec8ede Add version 5.4.2 constant
This commit adds the version 5.4.2 constant to master.
2017-06-01 11:25:19 -04:00
Jason Tedor 4185337df1 Add version 5.3.3 constant
This commit adds the version 5.3.3 constant to master.
2017-06-01 11:18:25 -04:00
Jay Modi 7526c29a05 Provide the TransportRequest during validation of a search context (#24985)
This commit provides the TransportRequest that caused the retrieval of a search context to the
SearchOperationListener#validateSearchContext method so that implementers have access to the
request.
2017-06-01 07:49:58 -06:00
Jason Tedor 9b4a189147 Add purge option to remove plugin CLI
By default, the remove plugin CLI command preserves configuration
files. This is so that if a user is upgrading the plugin (which is done
by first removing the old version and then installing the new version)
they do not lose their configuration file. Yet, there are circumstances
where preserving the configuration file is not desired. This commit adds
a purge option to the remove plugin CLI command.

Relates #24981
2017-06-01 08:53:39 -04:00
Boaz Leskes 1775e4253e Introducing a translog deletion policy (#24950)
Currently, the decisions regarding which translog generation files to delete are hard coded in the interaction between the `InternalEngine` and the `Translog` classes. This PR extracts it to a dedicated class called `TranslogDeletionPolicy`, for two main reasons:

1) Simplicity - the code is easier to read and understand (no more two phase commit on the translog, the Engine can just commit and the translog will respond)
2) Preparing for future plans to extend the logic we need - i.e., retain multiple lucene commit and also introduce a size based retention logic, allowing people to always keep a certain amount of translog files around. The latter is useful to increase the chance of an ops based recovery.
2017-06-01 14:04:21 +02:00
Thomas Decaux 3eabb3acfd Enforce validation for PathHierarchy tokenizer (#23510)
If delimiter or replacement parameter are an empty string, the error is not clear enough to indicate how to fix it.
With this change, the user knows these parameter must be a non empty string.
2017-06-01 12:54:16 +02:00
Tim Brooks 0424099674 Fix broken build from stream with zero bytes (#24993)
This is related to #24927. There was a small possibility that a test
was attempting to compress a stream with zero bytes. This was causing
a failure.

This test now requires at least one byte.
2017-05-31 17:33:11 -05:00
Tim Brooks 90a5574c93 Add CompressibleBytesOutputStream for compression (#24927)
This is a follow-up to #23941. Currently there are a number of
complexities related to compression. The raw DeflaterOutputStream must
be closed prior to sending bytes to ensure that EOS bytes are written.
But the underlying ReleasableBytesStreamOutput cannot be closed until
the bytes are sent to ensure that the bytes are not reused.

Right now we have three different stream references hanging around in
TCPTransport to handle this complexity. This commit introduces
CompressibleBytesOutputStream to be one stream implemenation that will
behave properly with or without compression enabled.
2017-05-31 11:00:40 -05:00
Lee Hinman 9d6cb4cb6d Remove unused MeterMetric and specialized EWMA (#24975)
This metric is not used in the ES codebase at all. It's also not as likely to be
used since it relies on a periodic "tick", which we don't currently use.
2017-05-31 09:05:22 -06:00
Jim Ferenczi ec64c2c05f Compute the took time of the query after the expand phase (#24902)
The took time computed for search requests does not take in account the expand search phase.
This change delays the computation to after the expand phase finishes.

Relates #24900
2017-05-31 12:42:05 +02:00
Masaru Hasegawa a77b38cdd1 Fix context suggester to read values from keyword type field (#24200)
Closes #24129
2017-05-31 11:35:01 +02:00
Martijn van Groningen 258be2b135
Moved `keyword_marker`, `trim`, `snowball` and `porter_stemmer` tokenfilter factories from core to common-analysis module.
Relates to #23658
2017-05-31 09:34:08 +02:00
Martijn van Groningen a089dc9dcd
Added more unit test coverage for terms aggregation and
removed terms agg integration tests that were replaced by unit tests.
2017-05-31 09:30:10 +02:00
Martijn van Groningen 9531ef25ec
Move OldIndexBackwardsCompatibilityIT#assertBasicSearchWorks over to full cluster restart qa module.
Relates to #24939
2017-05-31 09:27:41 +02:00
Tanguy Leroux 8e0d6015f9 [Test] Mute SearchResponseTests.testFromXContent()
And also AggregationsTests.testFromXContent() until
https://github.com/elastic/elasticsearch/pull/24964 is merged.
2017-05-31 09:16:23 +02:00
Adrien Grand 36a180ec20 Eliminate array access in tight loops when profiling is enabled. (#24959)
This makes profiling classes acquire a timer up-front that can be then reused
across all calls, in order to save bound checks for methods that are called in
tight loops.
2017-05-31 09:11:00 +02:00
Adrien Grand 71264c6239 PatternAnalyzer should lowercase wildcard queries when `lowercase` is true. (#24967) 2017-05-31 09:09:53 +02:00
Ryan Ernst 7c1211d2ed Scripting: Add StatefulFactoryType as optional intermediate factory in script contexts (#24974)
ScriptContexts currently understand a FactoryType that can produce
instances of the script InstanceType. However, for search scripts, this
does not work as we have the concept of LeafSearchScript that is created
per lucene segment. This commit effectively renames the existing
SearchScript class into SearchScript.LeafFactory, which is a new,
optional, class that can be defined within a ScriptContext.
LeafSearchScript is effectively renamed back into SearchScript. This
change allows the model of stateless factory -> stateful factory ->
script instance to continue, but in a generic way that any script
context may take advantage of.

relates #20426
2017-05-30 16:32:14 -07:00
Jason Tedor ac94253dce Clarify acquiring index shard permit
In previous work, we refactored the delay mechanism in index shard
operation permits to allow for async delaying of acquisition. This
refactoring made explicit when permit acquisition is disabled whereas
previously we were relying on an implicit condition, namely that all
permits were acquired by the thread trying to delay acquisition. When
using the implicit mechanism, we tried to acquire a permit and if this
failed, we returned a null releasable as an indication that our
operation should be queued. Yet, now we know when we are delayed and we
should not even try to acquire a permit. If we try to acquire a permit
and one is not available, we know that we are not delayed, and so
acquisition should be successful. If it is not successful, something is
deeply wrong. This commit takes advantage of this refactoring to
simplify the internal implementation.

Relates #24971
2017-05-30 16:22:17 -04:00
Jason Tedor b28141a990 Fill gaps on primary promotion
When a primary is promoted, it could have gaps in its history due to
concurrency and in-flight operations when it was serving as a
replica. This commit fills the gaps in the history of the promoted shard
after all operations from the previous term have drained, and future
operations are blocked. This commit does not handle replicating the
no-ops that fill the gaps to any remaining replicas, that is the
responsibility of the primary/replica sync that we are laying the ground
work for.

Relates #24945
2017-05-30 13:19:44 -04:00
Jim Ferenczi ce7195d81a Terms aggregation should remap global ordinal buckets when a sub-aggregator is used to sort the terms (#24941)
`terms` aggregations at the root level use the `global_ordinals` execution hint by default.
When all sub-aggregators can be run in `breadth_first` mode the collected buckets for these sub-aggs are dense (remapped after the initial pruning).
But if a sub-aggregator is not deferrable and needs to collect all buckets before pruning we don't remap global ords and the aggregator needs to deal with sparse buckets.
Most (if not all) aggregators expect dense buckets and uses this information to allocate memories.
This change forces the remap of the global ordinals but only when there is at least one sub-aggregator that cannot be deferred.

Relates #24788
2017-05-30 19:13:07 +02:00
Jason Tedor ddbc4687f6 Introduce clean transition on primary promotion
This commit introduces a clean transition from the old primary term to
the new primary term when a replica is promoted primary. To accomplish
this, we delay all operations before incrementing the primary term. The
delay is guaranteed to be in place before we increment the term, and
then all operations that are delayed are executed after the delay is
removed which asynchronously happens on another thread. This thread does
not progress until in-flight operations that were executing are
completed, and after these operations drain, the delayed operations
re-acquire permits and are executed.

Relates #24925
2017-05-30 11:39:36 -04:00
Jason Tedor 15fc71249c Fix typo in comment in ReplicationOperation.java
Within two lines of each other appears "fallthrough" and "fall through",
both typed by the same person who should have been paying better
attention and only one of these is correct and the inconsistency is
bothersome. This commit fixes the errant one.
2017-05-30 11:16:32 -04:00
Lee Hinman 0b3be42c10 Prevent Index & Delete request primaryTerm getter/setter, setShardId setter 2017-05-30 08:48:35 -06:00
Nik Everett 6d9ce957d4 Drop name from TokenizerFactory (#24869)
Drops `TokenizerFactory#name`, replacing it with
`CustomAnalyzer#getTokenizerName` which is much better targeted at
its single use case inside the analysis API.

Drops a test that I would have had to refactor which is duplicated by
`AnalysisModuleTests`.

To keep this change from blowing up in size I've left two mostly
mechanical changes to be done in followups:
1. `TokenizerFactory` can now be entirely dropped and replaced with
`Supplier<Tokenizer>`.
2. `AbstractTokenizerFactory`'s ctor still takes a `String` parameter
where the name once was.
2017-05-30 10:39:22 -04:00
Zachary Tong b8d7b83f8e Correctly set doc_count when MovAvg "predicts" values on existing buckets (#24892)
If the bucket already exists, due to non-overlapping series or missing data, the
MovAvg creates a merged bucket with the existing aggs + the new prediction.  This
fixes a small bug where the doc_count was not being set correctly.

Relates to #24327
2017-05-30 10:36:22 -04:00
Jason Tedor 9957bdf0ad Handle primary failure handling replica response
Today if the primary throws an exception while handling the replica
response (e.g., because it is already closed while updating the local
checkpoint for the replica), or because of a bug that causes an
exception to be thrown in the replica operation listener, this exception
is caught by the underlying transport handler plumbing and is translated
into a response handler failure transport exception that is passed to
the onFailure method of the replica operation listener. This causes the
primary to turn around and fail the replica which is a disastrous and
incorrect outcome as there's nothing wrong with the replica, it is the
primary that is broken and deserves a paddlin'. This commit handles this
situation by failing the primary.

Relates #24926
2017-05-30 10:05:11 -04:00
Boaz Leskes efcbfb7c32 testConcurrentWriteViewsAndSnapshot shouldn't flush concurrently
Fixes #24933
2017-05-30 14:22:23 +02:00
Jim Ferenczi fc35d51c3c [TEST] Fix FieldSortIT failures 2017-05-30 11:06:54 +02:00
Jim Ferenczi 628dabd663 Fix script field sort returning Double.MAX_VALUE for all documents (#24942)
This change fixes the script field sort when the returned type is a number.

Closes #24940
2017-05-30 09:23:04 +02:00
Jason Tedor b2abdd1174 Add a second refresh to concurrent relocation test
This commit adds a second refresh to the concurrent relocation
test. This is necessary as the first refresh might have brought back a
local checkpoint for a shard that a newly relocated primary became aware
of but did not yet receive a local checkpoint for that shard. When that
local checkpoint arrives on the new primary, the global checkpoint could
advance again and so we need a second replication action to push that
global checkpoint back out to the replica. This is indeed a hack, and it
will eventually be removed.

Closes #24599
2017-05-29 11:37:27 -04:00
Boaz Leskes dfdf496c1a Move the IndexDeletionPolicy to be engine internal (#24930)
The `IndexDeletionPolicy` is currently instantiated by `IndexShard` and is then passed through to the engine as a parameter. That's a shame as it is really just an implementation detail and the engine already has a method to acquire a commit.

This is preparing for a follow up PR that will we connect the index deletion policy with a new translog deletion policy.

Relates to #10708
2017-05-29 15:56:30 +02:00
Christoph Büscher 5741005812 [Tests] Harden InternalExtendedStatsTests (#24934)
The order in which double values are added in Java can give different results,
so in testing the sum and sumOfSquares we need to allow some delta for testing
equality. The difference can be larger for large sum values, so we should
account for this by making the delta in the assertion depend on the values
magnitude.

Closes #24931
2017-05-29 15:45:51 +02:00
Luca Cavanna 12654384a5 Add fromXContent method to ClearScrollResponse (#24909)
ClearScrollResponse can print out its content into an XContentBuilder as it implements ToXContentObject. This PR add a fromXContent method to it so that we are able to recreate the response object when parsing the response back. This will be used in the high level REST client.
2017-05-29 11:00:54 +02:00
Luca Cavanna ea00d343ae ClearScrollRequest to implement ToXContentObject (#24907)
ClearScrollRequest can be created from a request body, but it doesn't support the opposite, meaning printing out its content to an XContentBuilder. This is useful to the high level REST client and allows for better testing of what we parse.

Moved parsing method from RestClearScrollAction to ClearScrollRequest so that fromXContent and toXContent sit close to each other. Added unit tests to verify that body parameters override query_string parameters when both present (there is already a yaml test for this but unit test is even better)
2017-05-29 11:00:20 +02:00
Luca Cavanna 7be5758e6d SearchScrollRequest to implement ToXContentObject (#24906)
SearchScrollRequest can be created from a request body, but it doesn't support the opposite, meaning printing out its content to an XContentBuilder. This is useful to the high level REST client and allows for better testing of what we parse.

Moved parsing method from RestSearchScrollAction to SearchScrollRequest so that fromXContent and toXContent sit close to each other. Added unit tests to verify that body parameters override query_string parameters when both present (there is already a yaml test for this but unit test is even better)
2017-05-29 10:56:35 +02:00
mashudong 678730107c Fix bug in weight computation for query cache
When proportioning the shared RAM bytes across the shards of the query
cache, there's a computation that shares these bytes according to the
relative size of the shard cache to the total size of all the shard
caches. This computation had a bug where integer division was performed
instead which leads to this computation often being zero. This commit
fixes this bug by casting the numerator to a double before doing the
division so that double division is performed.

Relates #24856
2017-05-28 13:36:47 -04:00
Jason Tedor 09dd03e19f Verify Lucene version constants
The Lucene version constants for 5.4.1 and 5.5.0 are wrong, they are
listed as 6.5.0 instead of 6.5.1. This commit fixes these issues, and
adds a test to ensure that this does not happen again.

Relates #24923
2017-05-27 15:46:16 -04:00
Jason Tedor 3448028af7 Avoid double decrement on current query counter
This commit fixes a double decrement bug on the current query
counter. The double decrement arises in a situation when the fetch phase
is inlined for a query that is only touching one shard. After the query
phase succeeds we decrement the current query counter. If the fetch
phase ultimately fails, an exception is thrown and we decrement the
current query counter again in the catch block. We also add assertions
that all current stats counters remain non-negative at all
times.

Relates #24922
2017-05-27 15:40:37 -04:00
Nik Everett 5da8ce8318 Remove the need for _UNRELEASED suffix in versions (#24798)
Removes the need for the `_UNRELEASED` suffix on versions by detecting if a version should be unreleased or not based on the versions around it. This should make it simpler to automate the task of adding a new version label.
2017-05-26 18:36:32 -04:00
Lee Hinman 23fb36cc87 Adjust available and free bytes to be non-negative on huge FSes
In #23093 we made a change so that total bytes for a filesystem would not be a
negative value when the total bytes were > Long.MAX_VALUE.

This fixes #24453 which had a related issue where `available` and `free` bytes
could also be so large that they were negative. These will now return
`Long.MAX_VALUE` for the bytes if the JDK returns a negative value.
2017-05-26 12:59:53 -06:00
Nik Everett e072cc7770 Begin replacing static index tests with full restart tests (#24846)
These tests spin up two nodes of an older version of Elasticsearch,
create some stuff, shut down the nodes, start the current version,
and verify that the created stuff works.

You can run `gradle qa:full-cluster-restart:check` to run these
tests against the head of the previous branch of Elasticsearch
(5.x for master, 5.4 for 5.x, etc) or you can run
`gradle qa:full-cluster-restart:bwcTest` to run this test against
all "index compatible" versions, one after the other. For master
this is every released version in the 5.x.y version *and* the tip
of the 5.x branch.

I'd love to add more to these tests in the future but these
currently just cover the functionality of the `create_bwc_index.py`
script and start to cover the assertions in the
`OldIndexBackwardsCompatibilityIT` test.
2017-05-26 14:07:48 -04:00
Jim Ferenczi 47cf7825dd Move BWC version to 5.5 after backport
Relates to #24517
2017-05-26 14:57:07 +02:00
Jim Ferenczi 9ef414fead Merge branch 'mattweber-multiple_collapse_inner_hits' 2017-05-26 13:28:08 +02:00
Matt Weber 601a61a91c Support Multiple Collapse Inner Hits
Support multiple named inner hits on a field collapsing
request.
2017-05-26 13:23:57 +02:00
Ryan Ernst 74e031e842 Scripting: Rename CompiledType to FactoryType in ScriptContext (#24897)
This commit renames the concept of the "compiled type" to a "factory
type", along with all implementations of this class to be named Factory.
This brings it inline with the classes purpose.
2017-05-26 00:02:54 -07:00
Ryan Ernst 8eab1fefa1 Scripting: Make contexts available to ScriptEngine construction (#24896)
This commit adds collection of all contexts to the parameters of
getScriptEngine. This will allow script engines like painless to
precache extra information about the contexts.
2017-05-25 16:55:47 -07:00
Jason Tedor 9a118691bf Mute index and relocate concurrently
This test is failing sporadically and for now we mute it as we have a
failure with additional logging that should hopefully enable us to
assess the situation.
2017-05-25 17:37:36 -04:00
Ryan Ernst 8aaea51a0a Scripting: Move context definitions to instance type classes (#24883)
This is a simple refactoring to move the context definitions into the
type that they use. While we have multiple context names for the same
class at the moment, this will eventually become one ScriptContext per
instance type, so the pattern of a static member on the interface called
CONTEXT can be used. This commit also moves the consolidated list of
contexts provided by core ES into ScriptModule.
2017-05-25 12:18:45 -07:00
Jay Modi f60f79f361 Put mapping and index template requests do not need content type detection for 5.3.0+ (#24835)
This change cleans up some missed TODOs for content type detection on the source of put mapping and
put index template requests. In 5.3.0 and newer versions, the source is always JSON so the content
type detection is not needed. The TODOs were missed after the change was backported to 5.3.

Relates #24798
2017-05-25 12:51:44 -06:00
Jay Modi fab6b00ca7 Add the ability to store objects with a ScrollContext (#24777)
This commit adds the ability to store and retrieve data that should be associated with a
ScrollContext. Additionally the ScrollContext was made final as we should only have a single
implementation of this concept.
2017-05-25 12:30:00 -06:00
Simon Willnauer 1325681a03 Fix error message if an incompatible node connects (#24884)
This message broken in recent refactoring, this commit also adds a
basic unit-test to ensure we maintain the correct version.
2017-05-25 15:02:43 +02:00
Ryan Ernst 7d03cff820 Scripting: Make ScriptEngine.compile generic on the script context (#24873)
This commit changes the compile method of ScriptEngine to be generic in
the same way it is on ScriptService. This moves the shim of handling the
two existing context classes into each script engine, so that each
engine can be worked on independently to convert to real handling of
contexts.
2017-05-24 20:06:32 -07:00
Ryan Ernst 5581a0b2f0 Scripting: Remove unnecessary generic type from ScriptContext (#24877)
When developing the new ScriptContext, the compiled type was original
generic, so that the instance type was also necessary. However, since
CompiledType is all that is used by the compile method signature, we
actually don't need the instance type to be generic. This commit removes
the InstanceType, and finds the Class for it through reflection on the
CompiledType method.
2017-05-24 19:20:49 -07:00
Ryan Ernst 1daacd97b0 Scripting: Add instance and compiled classes to script contexts (#24868)
This commit modifies the compile method of ScriptService to be context
aware. The ScriptContext is now a generic class which contains both the
instance type and compiled type for a script. Instance type may be
stateful (for example, pre loading field information for the index a
script will execute on, like in expressions), while the compiled type is
stateless and used to construct instance type instances. This change is
only a first step to cutover ScriptService to the new paradigm. It only
converts callers to the script service, and has a small shim to wrap
compilation from the script engines to support the current two fixed
instance types, SearchScript and ExecutableScript.
2017-05-24 14:29:02 -07:00
Ryan Ernst 0ddd219423 Scripting: Add default implementation of close() for ScriptEngine (#24851)
Since groovy was removed, we no longer have any ScriptEngines with
resources to release. We may want to keep the option open for a script
engine to close resources, but this would not be common. This commit
adds a default implementation to ScriptEngine for `close()` to reduce
the boiler plate that must be added for a ScriptEngine implementation.
2017-05-24 13:19:27 -07:00
Lee Hinman 7a6db074ee [TEST] Add test for retrying replica operations with real network
Related to #24745
2017-05-24 13:31:30 -06:00
Jason Tedor 0499849b1b Increase logging in concurrent relocation test
This commit increases the logging level on the index and relocate
concurrently test to obtain some insight into the global checkpoint
moving backwards.
2017-05-24 11:51:00 -04:00
Boaz Leskes b58f198d94 NoMasterNodeIT shouldn't try to validate the length of a timeout
The current log tries make sure we waited some (but not too long). This is unpredictable and fails all the time. This commit removes all of it and just make sure that we throw the right exceptions after timing out.

Fixes #24369
2017-05-24 16:15:15 +02:00
Simon Willnauer aa0b9f9b39 Remote duplicate test from RemoteClusterServiceTests
The test `testEnsureConnected` was accidentially introduces during #24845
2017-05-24 16:04:25 +02:00
Ali Beyad 877f2d671b [TEST] adds TRACE logging to org.elasticsearch.snapshots for the
SharedClusterSnapshotRestoreIT#testDataFileCorruptionDuringRestore test
2017-05-24 09:50:17 -04:00
Ali Beyad 68f03b88c4 [TEST] increases testDataFileCorruptionDuringRestore logging to TRACE to
help debug sporadic tests failures.
2017-05-24 09:38:54 -04:00
markharwood b7197f5e21 SignificantText aggregation - like significant_terms, but for text (#24432)
* SignificantText aggregation - like significant_terms but doesn’t require fielddata=true, recommended used with `sampler` agg to limit expense of tokenizing docs and takes optional `filter_duplicate_text`:true setting to avoid stats skew from repeated sections of text in search results.

Closes #23674
2017-05-24 13:46:43 +01:00
Yannick Welsch b5adb3cce9 Remove AlreadyExpiredException (#24857)
This is a relict from the TTL functionality that has been removed in #21670
2017-05-24 13:49:02 +02:00
Boaz Leskes 6bc5b1dbcd Guarantee that translog generations are seqNo conflict free (#24825)
With #24779 in place, we can now guaranteed that a single translog generation file will never have a sequence number conflict that needs to be resolved by looking at primary terms. These conflicts can a occur when a replica contains an operation which isn't part of the history of a newly promoted primary. That primary can then assign a different operation to the same slot and replicate it to the replica.

PS. Knowing that each generation file is conflict free will simplifying repairing these conflicts when we read from the translog.

PPS. This PR also fixes some bugs in the piping of primary terms in the bulk shard action. These bugs are a result of the legacy of IndexRequest/DeleteRequest being a ReplicationRequest. We need to change that as a follow up.

Relates to #10708
2017-05-24 13:26:39 +02:00
Simon Willnauer ac6a6d6fe8 Ensure remote cluster is connected before fetching `_field_caps` (#24845)
If a cluster disconnects and comes back up we should ensure that
we connected to the cluster before we fire the requests.

Closes #24763
2017-05-24 13:15:28 +02:00
Ryan Ernst bf49d37ab3 Test: Convert test script engine impls to use MockScriptEngine (#24854)
This commit cleans up tests which currently use custom script engine
implementations, converting them to use a MockScriptEngine with script
functions provided by the tests. It also creates a common set of metric
scripts which were copied across a couple metric agg tests.
2017-05-23 20:34:12 -07:00
Jim Ferenczi 4707377cea Move InnerHitBuilder queries BWC version to 5.5 after the backport
Relates #24676
2017-05-23 22:41:39 +02:00
Christoph Büscher 24a8ba5ca8 Use ParseField constants in ParsedGeoBounds (#24849) 2017-05-23 20:15:40 +02:00
Lee Hinman 82a369737b [TEST] Add additional assert logging 2017-05-23 10:37:40 -06:00
markharwood c71ae3519f Search: Fairer balancing when routing searches by session ID (#24671)
A user reported uneven balancing of load on nodes handling search requests from Kibana which supplies a session ID in a routing preference. Each shardId was selecting the same node for a given session ID because one data node had all primaries and the other data node held all replicas after cluster startup.
This change counteracts the tendency to opt for the same node given the same user-supplied preference by incorporating shard ID in the hash of the preference key. This will help randomise node choices across shards.

Closes #24642
2017-05-23 14:43:44 +00:00
Boaz Leskes b2ccb6b0a8 simplify flow/stop control in testConcurrentWriteViewsAndSnapshot
Thread starvation of the control thread could cause the writer to keep on generating ops and make the test go out of control (OOM).
2017-05-23 15:40:53 +02:00
Jason Tedor 6ec485d30d Suppress inspections on assertions helper
This commit adds comments to org.elasticsearch.Assertions that disables
IntelliJ from complaining about using assert with side-effects, and
using constant conditions there as the side-effect with a constant
condition is intentionally employed.
2017-05-23 09:34:51 -04:00
Jason Tedor c179c6a4c9 Add assertions enabled helper
Today in the code base we have lots of ugly code blocks like:

  boolean assertionsEnabled = false;
  assert assertionsEnabled = true;
  if (assertionsEnabled) {
    // something
  }

These are a nuisance. Instead, we can do this in exactly one place and
replace these blocks with

  if (Assertions.ENABLED) {
    // something
  }

The cool thing here is that since this is a static final field, the JIT
can optimize away the check at runtime if assertions are disabled.

Relates #24834
2017-05-23 08:22:18 -04:00
Jim Ferenczi 9087803cd9 Add the ability to define custom inner hit sub context builder (#24676)
This commit moves the handling of nested and parent/child inner hits to specialized classes that can be defined outside of ES core.
InnerHitBuilderContext is now used by the parent query (nested or hasChild, ...) to build the sub context from the InnerHitBuilder definition.
BWC is also ensured so that nodes in previous versions can still send/receive inner hits to/from this version.

Relates #20257
2017-05-23 13:06:22 +02:00
Boaz Leskes a5fffa2988 relax the assertion about the existence of MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID in the user commit data
Since #24149 we started storing the maxUnsafeAutoIdTimestamp in the lucene commit. The assertions in that PR were built on the assumption that recovering a primary always ends up committing lucene. That is sadly [not true](a72eaa8e0f/core/src/main/java/org/elasticsearch/index/engine/InternalEngine.java (L313)).

Closes #24809
2017-05-23 09:08:30 +02:00
Ali Beyad d3c95b8d9d Removes the 5.3.3 version constant (#24833)
After releasing 5.3.2, the 5.3.3 version constant was created.  However,
this causes issues for the rolling upgrade tests, which expect to have
all older versions artifacts published and no point releases created off
of the older versions (older meaning more than one version behind the
current version).  This commit removes the 5.3.3 version constant,
assuming we will not need it anywhere.
2017-05-22 21:53:16 -04:00
Jack Conradson 8887bcc4c6 Fix settings names for script.allowed_types and script.allowed_contexts. (#24831)
Fixes #24830
2017-05-22 15:08:45 -07:00
Ryan Ernst 52d504bb5f Scripting: Simplify ScriptContext (#24818)
As we work towards contexts implying the return type of compilation, we
first need ScriptContext to not be an enum. This commit removes the
Standard enum and Plugin subclass of ScriptContext.
2017-05-22 13:11:15 -07:00
Nicholas Knize 5ae48f7dd5 Fix RangeFieldMapper rangeQuery to properly handle relations
This commit fixes the RangeFieldMapper and RangeQueryBuilder to pass the correct relation to the RangeQuery when performing a range query over range fields.
2017-05-22 11:37:06 -05:00
javanna 7a3e38eb8e Merge branch 'master' into feature/client_aggs_parsing 2017-05-22 12:25:14 +02:00
Luca Cavanna 726e6c45ab Move getType to Aggregation interface (#24822)
Given that both InternalAggregation and ParsedAggregation have this method, it makes sense to move it to the interface they both implement.
2017-05-22 10:49:23 +02:00
olcbean e08e92d934 Deleting a document from a non-existing index creates the should not auto create it, unless using EXTERNAL* versioning (#24518)
Currently a `delete document` request against a non-existing index actually **creates** this index.

With this change the `delete document` no longer creates the previously non-existing index and throws an `index_not_found` exception instead.

However as discussed in https://github.com/elastic/elasticsearch/pull/15451#issuecomment-165772026, if an external version is explicitly used, the current behavior is preserved and the index is still created and the document is marked for deletion.

Fixes #15425
2017-05-22 10:00:22 +03:00