Commit Graph

3198 Commits

Author SHA1 Message Date
Igor Motov d340c190b2
Replace List with Map in PersistentTasksInProgress
Store currently running persistent tasks in a map instead of a list.

Original commit: elastic/x-pack@f88c9adef5
2018-01-31 11:58:06 +01:00
David Kyle 32e406181e
Fix check style error after upgrade
Original commit: elastic/x-pack@3bf4025f78
2018-01-31 11:58:06 +01:00
Igor Motov ac67d02bc3
Add support for task status on persistent tasks
Similarly to task status on normal tasks it's now possible to update task status on the persistent tasks. This should allow updating the state of the running tasks (such as loading, started, etc) as well as store intermediate state or progress.

Original commit: elastic/x-pack@048006b467
2018-01-31 11:58:06 +01:00
Martijn van Groningen 777b21f2ef
Add a number of auxiliary methods to persistent tasks classes.
Original commit: elastic/x-pack@7f44b41b7a
2018-01-31 11:57:02 +01:00
Igor Motov f136bfa6e0
Adds support for persistent actions
A persistent action is a transport-like action that is using the cluster state instead of transport to start tasks. This allows persistent tasks to survive restart of executing nodes. A persistent action can be implemented by extending TransportPersistentAction. TransportPersistentAction will start the task by using PersistentActionService, which controls persistent tasks lifecycle.  See TestPersistentActionPlugin for an example implementing a persistent action.

Original commit: elastic/x-pack@5e83f1bfa3
2018-01-31 11:08:56 +01:00
Jim Ferenczi cb1fef7f6e
Fix intermittent failure in InternalEngineTest#testRefreshScopedSearcher (#28417)
This change switches the merge policy to none (for this specific test) in order to make sure that refreshes are always triggered
 by a change in the writer.

 Closes #27514
2018-01-31 09:24:15 +01:00
Nik Everett 3b6af15a60
XContent: Factor deprecation handling into callback (#28449)
Factors the way in which XContent parsing handles deprecated fields
into a callback that is set at parser construction time. The goals here
are:
1. Remove Log4J as a dependency of XContent so that XContent can be used
by clients without forcing log4j and our particular deprecation handling
scheme.
2. Simplify handling of deprecated fields in tests. Now tests can listen
directly for the deprecation callback rather than digging through a
ThreadLocal.

More accurately, this change begins this work. It deprecates a number of
methods, pointing folks to the new versions of those methods that take
`DeprecationHandler`. The plan is to slowly drop these deprecated
methods. Once they are entirely removed we can remove Log4j as
dependency of XContent.
2018-01-30 18:21:10 -05:00
Simon Willnauer 3bf8554114
Remove tribe node support (#28443)
Tribe node has been superseeded by Cross-Cluster-Search. This change
removes the tribe node support entirely.
2018-01-30 20:40:19 +01:00
Alexander Reelsen 1d311dfb65 Versions: Add 6.1.4/5.6.8 snapshot versions 2018-01-30 20:13:34 +01:00
Christoph Büscher 6731c76900
Add ranking evaluation API to High Level Rest Client (#28357)
This change adds support for the new ranking evaluation API to the High Level Rest Client.
This mostly means adding support for parsing the various response objects back from the
REST representation. It includes one change to the response syntax where previously we didn't
print the type of the metric details section but we now need it to pick the right parser to
parse this section back.

Closes #28198
2018-01-30 17:48:09 +01:00
Luca Cavanna 2c99bfc947
REST high-level client: Fix parsing of script fields (#28395)
Script fields can get a bit more complicated than just stored fields. A script can return null, an object and also an array. Extended parsing to support such valid values. Also renamed util method from `parseStoredFieldsValue` to `parseFieldsValue` given that it can parse stored fields but also script fields, anything that's returned as `fields`.

Closes #28380
2018-01-30 13:19:08 +01:00
Boaz Leskes 613fc1654f testTranslogReplayWithFailure: checkStyle 2018-01-30 11:26:06 +01:00
Boaz Leskes f5f679f921 testTranslogReplayWithFailure: do not assume engine is recoverable when disk errors stop
The test currently makes the assumption that if underlying directory stops throwing exceptions, we can always open the engine. This is not the case as some errors can cause a corruption marker to be placed in the store.

This commit refactors the test to only check that everything is OK if the engine was successfully opened. On top of that, there is no point in checking replay with no errors as we have another test for that.

Closes #28426
2018-01-30 10:02:26 +01:00
Alan Woodward 424ecb3c7d
Add ability to index prefixes on text fields (#28290)
This adds the ability to index term prefixes into a hidden subfield, enabling prefix queries to be run without multitermquery rewrites. The subfield reuses the analysis chain of its parent text field, appending an EdgeNGramTokenFilter. It can be configured with minimum and maximum ngram lengths. Query terms with lengths outside this min-max range fall back to using prefix queries against the parent text field.

The mapping looks like this:

"my_text_field" : {
"type" : "text",
"analyzer" : "english",
"index_prefix" : { "min_chars" : 1, "max_chars" : 10 }
}

Relates to #27049
2018-01-30 08:26:56 +00:00
Yannick Welsch 6f84503c33
Use final fields in UnicastZenPing request/response objects (#28406)
Prevents a NullPointerException that can happen due to concurrency in UnicastZenPing, see #21658.
2018-01-29 14:30:51 +01:00
Alan Woodward e208e959bd
TextFieldMapper defaults can be final (#28313) 2018-01-29 10:05:54 +00:00
Ryan Ernst b47b399f00
Settings: Reimplement keystore format to use FIPS compliant algorithms (#28255)
This commit switches the internal format of the elasticsearch keystore
to no longer use java's KeyStore class, but instead encrypt the binary
data of the secrets using AES-GCM. The cipher key is generated using
PBKDF2WithHmacSHA512. Tests are also added for backcompat reading the v1
and v2 formats.
2018-01-26 15:51:07 -08:00
Ryan Ernst 3dd833ca0a
Plugins: Use one confirmation of all meta plugin permissions (#28366)
Currently meta plugins will ask for confirmation of security policy
exceptions for each bundled plugin. This commit collects the necessary
permissions of each bundled plugin, and asks for confirmation of all of
them at the same time.
2018-01-26 15:44:44 -08:00
Nhat Nguyen 414b5de661 TEST: Lower num of shards in testShrinkIndexPrimaryTerm
In some cases testShrinkIndexPrimaryTerm creates then 'mutates' 210
shards. If each shard opens more than 10 files (translog, lucene index),
we exceeded the maximum allowed file handles. In our test, the number of
file handles is limited to 2048 by HandleLimitFS. This commit reduces
the number of shards in testShrinkIndexPrimaryTerm to avoid such errors.

Closes #28153
2018-01-26 17:16:00 -05:00
Nhat Nguyen 583085d7e8 Increase timeout for ensureGreen in testShrinkIndexPrimaryTerm
If we have created 210 shards, we may need more than 30 seconds for all
shards become green.

Relates #28153
2018-01-26 15:31:37 -05:00
Lee Hinman c25a4637e8 Ensure total nanosecond time for tasks is at least 1 nanosecond
In rare cases the total nanoseconds for an entire window of operations can be 0
nanoseconds, causing the assertion in
QueueResizingEsThreadPoolExecutor.calculateLambda to trip. This ensures that we
calculate the lambda value with at least 1 nanosecond.

Resolves #27607
2018-01-26 11:36:44 -07:00
Lee Hinman 96e7da53c7 [TEST] Expand assert to mention which files are not deleted
Relates to #25335
2018-01-26 09:03:51 -07:00
Antonio Matarrese f61591c6ec Fix string terms get key as number to see integers
Currently this method parses the string as a double. This means that it
might lose accuracy if the value is a long that is greater than
2^52. This commit changes this method to try to detect whether the
string represents a long first.
2018-01-26 06:22:42 -05:00
Nhat Nguyen f39402a039
Fix peer recovery flushing loop (#28350)
Today after writing an operation to an engine, we will call 
`IndexShard#afterWriteOperation` to flush a new commit if needed. The 
`shouldFlush` condition is purely based on the uncommitted translog size
and the translog flush threshold size setting. However this can cause a
replica execute an infinite loop of flushing in the following situation.

1. Primary has a fully baked index commit with its local checkpoint 
equals to max_seqno
2. Primary sends that fully baked commit, then replays all retained
translog operations to the replica
3. No operations are added to Lucence on the replica as seqno of these
operations are at most the local checkpoint
4. Once translog operations are replayed, the target calls 
`IndexShard#afterWriteOperation` to flush. If the total size of the
replaying operations exceeds the flush threshold size, this call will
`Engine#flush`. However the engine won't flush as its index writer does
not have any uncommitted operations. The method
`IndexShard#afterWriteOperation` will keep flushing as the condition
`shouldFlush` is still true.

This issue can be avoided if we always flush if the `shouldFlush` 
condition is true.
2018-01-25 14:29:46 -05:00
olcbean 9db23e48cd Add Indices Aliases API to the high level REST client (#27876)
Relates to #27205
2018-01-25 14:34:06 +01:00
olcbean 0c83240b5f Java Api clean up: remove deprecated `isShardsAcked` (#28311)
This PR removes previously deprecated `isShardsAcked()` method in
favour of `isShardsAcknowledged()` on `CreateIndexResponse`, `CreateIndexClusterStateUpdateResponse` and `RolloverResponse` 

Related to #27784
Follow-up of #27819
2018-01-25 14:13:20 +01:00
Martijn van Groningen dea60fed50
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  [Docs] Fix explanation for `from` and `size` example (#28320)
  Adapt bwc version after backport #28358
  Always return the after_key in composite aggregation response (#28358)
  Adds test name to MockPageCacheRecycler exception (#28359)
  Adds a note in the `terms` aggregation docs regarding pagination (#28360)
  [Test] Fix DiscoveryNodesTests.testDeltas() (#28361)
  Update packaging tests to work with meta plugins (#28336)
  Remove Painless Type from MethodWriter in favor of Java Class. (#28346)
  [Doc] Fixs typo in reverse-nested-aggregation.asciidoc (#28348)
  Reindex: Shore up rethrottle test
  Only assert single commit iff index created on 6.2
  isHeldByCurrentThread should return primitive bool
  [Docs] Clarify `html` encoder in highlighting.asciidoc (#27766)
  Fix GeoDistance query example (#28355)
  Settings: Introduce settings updater for a list of settings (#28338)
  Adapt bwc version after backport #28310
2018-01-25 13:44:08 +01:00
Jim Ferenczi 95c45aeb5d Adapt bwc version after backport #28358 2018-01-25 09:26:10 +01:00
Jim Ferenczi c26d4ac6c1
Always return the after_key in composite aggregation response (#28358)
This change adds the `after_key` of a composite aggregation directly in the response.
It is redundant when all buckets are not filtered/removed by a pipeline aggregation since in this case the `after_key` is always the last bucket
in the response. Though when using a pipeline aggregation to filter composite buckets, the `after_key` can be lost if the last bucket is filtered.
This commit fixes this situation by always returning the `after_key` in a dedicated section.
2018-01-25 09:15:27 +01:00
Tanguy Leroux 5f0cb3a07e
[Test] Fix DiscoveryNodesTests.testDeltas() (#28361)
The DiscoveryNodes.Delta was changed in #28197. Previous/Master nodes
are now always set in the `Delta` (before the change they were set only
if the master changed) and the `masterChanged()` method is now based on
object equality and nodes ephemeral ids (before the change it was based
on nodes id).

This commit adapts the DiscoveryNodesTests.testDeltas() to reflect the
changes.
2018-01-25 08:52:30 +01:00
Nhat Nguyen 7847cded80 Only assert single commit iff index created on 6.2
We introduced a single commit assertion when opening an index but create
a new translog. However, this assertion is not held in this situation.

1. A replica with two commits c1 and c2 starts peer-recovery with c1
2. The recovery is sequence-based recovery but the primary is before 6.2 so
it sent true for “createNewTranslog”
3. Replica opens engine and create translog. We expect "open index and
create translog" have 1 commit but we have c1 and c2.

This commit makes sure to assert this iff the index was created on 6.2+.
2018-01-24 10:49:44 -05:00
Nhat Nguyen 80a7943d6a isHeldByCurrentThread should return primitive bool 2018-01-24 10:48:05 -05:00
Alexander Reelsen a87714aafc
Settings: Introduce settings updater for a list of settings (#28338)
This introduces a settings updater that allows to specify a list of
settings. Whenever one of those settings changes, the whole block of
settings is passed to the consumer.

This also fixes an issue with affix settings, when used in combination
with group settings, which could result in no found settings when used
to get a setting for a namespace.

Lastly logging has been slightly changed, so that filtered settings now
only log the setting key.

Another bug has been fixed for the mock log appender, which did not
work, when checking for the exact message.

Closes #28047
2018-01-24 09:47:17 +01:00
Martijn van Groningen 5b2dceea74
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  Remove redundant argument for buildConfiguration of s3 plugin (#28281)
  Completely remove Painless Type from AnalyzerCaster in favor of Java Class. (#28329)
  Fix spelling error
  Reindex: Wait for deletion in test
  Reindex: log more on rare test failure
  Ensure we protect Collections obtained from scripts from self-referencing (#28335)
  [Docs] Fix asciidoc style in composite agg docs
  Adds the ability to specify a format on composite date_histogram source (#28310)
  Provide a better error message for the case when all shards failed (#28333)
  [Test] Re-Add integer_range and date_range field types for query builder tests (#28171)
  Added Put Mapping API to high-level Rest client (#27869)
  Revert change that does not return all indices if a specific alias is requested via get alias api. (#28294)
  Painless: Replace Painless Type with Java Class during Casts (#27847)
  Notify affixMap settings when any under the registered prefix matches (#28317)
2018-01-24 09:25:54 +01:00
Jim Ferenczi b10d166190 Adapt bwc version after backport #28310 2018-01-24 09:17:30 +01:00
Simon Willnauer 4d3f7a7695
Ensure we protect Collections obtained from scripts from self-referencing (#28335)
Self referencing maps can cause SOE if they are iterated ie. in their toString methods. This chance adds some protected to the usage of those collections.
2018-01-23 16:57:26 +01:00
Jim Ferenczi 19cfc25873
Adds the ability to specify a format on composite date_histogram source (#28310)
This commit adds the ability to specify a date format on the `date_histogram` composite source.
If the format is defined, the key for the source is returned as a formatted date.

Closes #27923
2018-01-23 15:14:49 +01:00
Simon Willnauer d31e964a86
Provide a better error message for the case when all shards failed (#28333)
Today we don't specify a cause which can make debugging very very tricky.
This change is best effort to supply at least one cause for the failure.
2018-01-23 14:50:02 +01:00
Christoph Büscher ba9e2e44cb
[Test] Re-Add integer_range and date_range field types for query builder tests (#28171)
The tests for those field types were removed in #26549 because the range mapper
was moved to a module, but later this mapper was moved back to core in #27854.
This change adds back those two field types like before to the general setup in
AbstractQueryTestCase and adds some specifics to the RangeQueryBuilder and
TermsQueryBuilder tests. Also adding back an integration test in SearchQueryIT that
has been removed before but that can be kept with the mapper back in core now.

Relates to #28147
2018-01-23 13:08:54 +01:00
Catalin Ursachi cf61d792b2 Added Put Mapping API to high-level Rest client (#27869)
Relates to #27205
2018-01-23 11:03:32 +01:00
Martijn van Groningen 4ef341a0c3
Revert change that does not return all indices if a specific alias is requested via get alias api. (#28294)
Reopens #27763
2018-01-23 09:06:02 +01:00
Lee Hinman ba5b583203
Notify affixMap settings when any under the registered prefix matches (#28317)
* Notify affixMap settings when any under the registered prefix matches

Previously if an affixMap setting was registered, and then a completely
different setting was applied, the affixMap update consumer would be notified
with an empty map. This caused settings that were previously set to be unset in
local state in a consumer that assumed it would only be called when the affixMap
setting was changed.

This commit changes the behavior so if a prefix `foo.` is registered, any
setting under the prefix will have the update consumer notified if there are
changes starting with `foo.`.

Resolves #28316

* Add unit test

* Address feedback
2018-01-22 11:55:54 -07:00
Jason Tedor 437459d2f9 Merge branch 'master' into ccr
* master:
  Trim down usages of `ShardOperationFailedException` interface (#28312)
  Do not return all indices if a specific alias is requested via get aliases api.
  [Test] Lower bwc version for rank-eval rest tests
  CountedBitSet doesn't need to extend BitSet. (#28239)
  Calculate sum in Kahan summation algorithm in aggregations (#27807) (#27848)
  Remove the `update_all_types` option. (#28288)
  Add information when master node left to DiscoveryNodes' shortSummary() (#28197)
  Provide explanation of dangling indices, fixes #26008 (#26999)
2018-01-22 11:50:16 -05:00
Luca Cavanna 0c83ee2a5d
Trim down usages of `ShardOperationFailedException` interface (#28312)
In many cases we use the `ShardOperationFailedException` interface to abstract an exception that can only be of one type, namely `DefaultShardOperationException`. There is no need to use the interface in such cases, the concrete type should be used instead. That has the additional advantage of simplifying parsing such exceptions back from rest responses for the high-level REST client
2018-01-22 15:51:46 +01:00
Martijn van Groningen 509ecf2aa6
Do not return all indices if a specific alias is requested via get aliases api.
If a get alias api call requests a specific alias pattern then
indices not having any matching aliases should not be included in the response.

Closes #27763
2018-01-22 14:02:53 +01:00
Adrien Grand 8d195c86de
CountedBitSet doesn't need to extend BitSet. (#28239) 2018-01-22 12:43:34 +01:00
kel 452c36c552 Calculate sum in Kahan summation algorithm in aggregations (#27807) (#27848) 2018-01-22 12:42:56 +01:00
Adrien Grand 700d9ecc95
Remove the `update_all_types` option. (#28288)
This option is not useful in 7.x since no indices may have more than one type
anymore.
2018-01-22 12:03:07 +01:00
Tanguy Leroux 119b1b5c2b
Add information when master node left to DiscoveryNodes' shortSummary() (#28197)
This commit changes `DiscoveryNodes.Delta.shortSummary()` in order to
add information to the summary when the master node left.
2018-01-22 09:52:57 +01:00
Martijn van Groningen 2f17f91680
Merge remote-tracking branch 'es/master' into ccr
* es/master: (38 commits)
  Build: Add pom generation to meta plugins (#28321)
  Add 6.3 version constant to master
  Minor improvements to translog docs (#28237)
  [Docs] Remove typo in painless-getting-started.asciidoc
  Build: Fix meta plugin usage in integ test clusters (#28307)
  Painless: Add spi jar that will be published for extending whitelists (#28302)
  mistyping in one of the highlighting examples comment -> content (#28139)
  Documents applicability of term query to range type (#28166)
  Build: Omit dependency licenses check for elasticsearch deps (#28304)
  Clean up commits when global checkpoint advanced (#28140)
  Implement socket and server ChannelContexts (#28275)
  Plugins: Fix meta plugins to install bundled plugins with their real name (#28285)
  Build: Fix meta plugin integ test installation (#28286)
  Modify Abstract transport tests to use impls (#28270)
  Fork Groovy compiler onto compile Java home
  [Docs] Update tophits-aggregation.asciidoc (#28273)
  Docs: match between snippet to its description (#28296)
  [TEST] fix RequestTests#testSearch in case search source is not set
  REST high-level client: remove index suffix from indices client method names (#28263)
  Fix simple_query_string on invalid input (#28219)
  ...
2018-01-22 09:13:59 +01:00
Jason Tedor ef76d99d86 Add 6.3 version constant to master
This commit adds the 6.3 version constant to the master branch after 6.2
was cut from 6.x.
2018-01-20 22:16:59 -05:00
Nhat Nguyen 9db9bd52f7
Clean up commits when global checkpoint advanced (#28140)
Today we keep multiple index commits based on the current global 
checkpoint, but only clean up unneeded index commits when we have a new 
index commit. However, we can release the old index commits earlier once
the global checkpoint has advanced enough. This commit makes an engine
revisit the index deletion policy whenever a new global checkpoint value
is persisted and advanced enough.

Relates #10708
2018-01-18 15:45:06 -05:00
Jim Ferenczi c38c12e3bf
Fix simple_query_string on invalid input (#28219)
This change converts any exception that occurs during the parsing of
a simple_query_string to a match_no_docs query (instead of a null query)
when leniency is activated.

Closes #28204
2018-01-18 10:49:34 +01:00
Jason Tedor 6b0036e0e1
Add client actions to action plugin
This commit adds an extension point for client actions to action
plugins. This is useful for plugins to expose the client-side actions
without exposing the server-side implementations to the client. The
default implementation, of course, delegates to extracting the
client-side action from the server-side implementation.

Relates #28280
2018-01-17 21:57:03 -05:00
Tony Zeng 1335232e6b Add toString() implementation for UpdateRequest (#27997) 2018-01-17 17:04:04 +00:00
Alexander Reelsen 707d8d6fe6
Dependencies: Update joda time to 2.9.9 (#28261) 2018-01-17 14:58:52 +01:00
David Turner 9bd7f2c65b
Improve wording in deprecation message (#28259) 2018-01-17 12:42:20 +00:00
Tanguy Leroux 6256c330c0 [Test] Wait for no relocating shards in indices.stats/13_fields tests
MixedClusterClientYamlTestSuiteIT sometimes fails when executing the
indices.stats/13_fields/* REST tests. It does not reproduce locally
but the execution logs show that it failed when a shard is relocating
during the set up execution. This commit change the set up so that it
now waits for all shards to be active before executing the tests.

closes #26732, #27146
2018-01-17 13:35:29 +01:00
olcbean b98514c6d9 Add Close Index API to the high level REST client (#27734)
Add support for _close endpoint to the high level REST client

Relates to #27205
2018-01-17 11:47:08 +01:00
Jason Tedor 47547613ee Merge branch 'master' into ccr
* master:
  Fix third-party audit tasks on JDK 8
  Remove duplicated javadoc `fieldType` param
  Handle 5.6.6 and 6.1.2 release
  Introduce multi-release JAR
  Move the multi-get response tests to server
  Require JDK 9 for compilation (#28071)
  Revert "[Docs] Fix Java Api index administration usage (#28133)"
  Revert "[Docs] Fix base directory to include for put_mapping.asciidoc"
  Added multi get api to the high level rest client.
  [Docs] Clarify numeric datatype ranges (#28240)
  [Docs] Fix base directory to include for put_mapping.asciidoc
  Open engine should keep only starting commit (#28228)
2018-01-16 23:25:17 -05:00
Lee Hinman f2cd580332 Remove duplicated javadoc `fieldType` param 2018-01-16 16:34:44 -07:00
Nik Everett 4ec0569a19 Handle 5.6.6 and 6.1.2 release
Add new version constants for 5.6.6 and 6.1.2 release.
2018-01-16 16:41:05 -05:00
Jason Tedor 045dd4ad48
Introduce multi-release JAR
This commit introduces the ability for the core Elasticsearch JAR to be
a multi-release JAR containing code that is compiled for JDK 8 and code
that is compiled for JDK 9. At runtime, a JDK 8 JVM will ignore the JDK
9 compiled classfiles, and a JDK 9 JVM will use the JDK 9 compiled
classfiles instead of the JDK 8 compiled classfiles. With this work, we
utilize the new JDK 9 API for obtaining the PID of the running JVM,
instead of relying on a hack.

For now, we want to keep IDEs on JDK 8 so when the build is in an IDE we
ignore the JDK 9 source set (as otherwise the IDE would give compilation
errors). However, with this change, running Gradle from the command-line
now requires JAVA_HOME and JAVA_9_HOME to be set. This will require
follow-up work in our CI infrastructure and our release builds to
accommodate this change.

Relates #28051
2018-01-16 15:10:29 -05:00
Jason Tedor e5a698447b Move the multi-get response tests to server
This test file was accidentally pushed to core instead of server. This
commit moves this file to its proper location.
2018-01-16 14:11:31 -05:00
Christoph Büscher 8a58df46f3 Revert "[Docs] Fix Java Api index administration usage (#28133)"
This reverts commit 67c1f1c856.
2018-01-16 17:31:11 +01:00
Martijn van Groningen 853f7e8780
Added multi get api to the high level rest client.
Relates to #27205
2018-01-16 17:27:02 +01:00
Nhat Nguyen 65e90079ad
Open engine should keep only starting commit (#28228)
Keeping unsafe commits when opening an engine can be problematic because
these commits are not safe at the recovering time but they can suddenly
become safe in the future. The following issues can happen if unsafe
commits are kept oninit.

1. Replica can use unsafe commit in peer-recovery. This happens when a
replica with a safe commit c1 (max_seqno=1) and an unsafe commit c2
(max_seqno=2) recovers from a primary with c1(max_seqno=1). If a new
document (seqno=2) is added without flushing, the global checkpoint is
advanced to 2; and the replica recovers again, it will use the unsafe
commit c2 (max_seqno=2 <= gcp=2) as the starting commit for sequenced
based recovery even the commit c2 contains a stale operation and the
document (with seqno=2) will not be replicated to the replica.

2. Min translog gen for recovery can go backwards in peer-recovery. This
happens when a replica with a safe commit c1 (local_checkpoint=1,
recovery_translog_gen=1) and an unsafe commit c2 (local_checkpoint=2,
recovery_translog_gen=2). The replica recovers from a primary, and keeps
c2 as the last commit, then sets last_translog_gen to 2. Flushing a new
commit on the replica will cause exception as the new last commit c3
will have recovery_translog_gen=1. The recovery translog generation of a
commit is calculated based on the current local checkpoint. The local
checkpoint of c3 is 1 while the local checkpoint of c2 is 2.

3. Commit without translog can be used for recovery. An old index, which
was created before multiple-commits is introduced (v6.2), may not have a
safe commit. If that index has a snapshotted commit without translog and
an unsafe commit, the policy can consider the snapshotted commit as a
safe commit for recovery even the commit does not have translog.

These issues can be avoided if the combined deletion policy keeps only
the starting commit onInit.

Relates #27804
Relates #28181
2018-01-16 08:37:42 -05:00
Martijn van Groningen f2093e7238
Merge remote-tracking branch 'es/master' into ccr
* es/master: (30 commits)
  [Docs] Fix Java Api index administration usage (#28133)
  Fix eclipse build. (#28236)
  Never return null from Strings.tokenizeToStringArray (#28224)
  Fallback to TransportMasterNodeAction for cluster health retries (#28195)
  [Docs] Changes to ingest.asciidoc (#28212)
  TEST: Update logging for testAckedIndexing
  [GEO] Add WKT Support to GeoBoundingBoxQueryBuilder
  Painless: Add whitelist extensions (#28161)
  Fix daitch_mokotoff phonetic filter to use the dedicated Lucene filter (#28225)
  Avoid doing redundant work when checking for self references. (#26927)
  Fix casts in HotThreads. (#27578)
  Ignore the `-snapshot` suffix when comparing the Lucene version in the build and the docs. (#27927)
  Allow update of `eager_global_ordinals` on `_parent`. (#28014)
  Fix NPE on composite aggregation with sub-aggregations that need scores (#28129)
  `MockTcpTransport` to connect asynchronously (#28203)
  Fix synonym phrase query expansion for cross_fields parsing (#28045)
  Introduce elasticsearch-core jar (#28191)
  #28218: Update the Lucene version for 6.2.0 after backport
  upgrade to lucene 7.2.1 (#28218)
  [Docs] Fix an error in painless-types.asciidoc (#28221)
  ...
2018-01-16 12:22:22 +01:00
Christoph Büscher 67c1f1c856
[Docs] Fix Java Api index administration usage (#28133)
The Java API documentation for index administration currenty is wrong because
the PutMappingRequestBuilder#setSource(Object... source) and
CreateIndexRequestBuilder#addMapping(String type, Object... source) methods
delegate to methods that check that the input arguments are valid key/value
pairs:

https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-admin-indices.html

This changes the docs so the java api code examples are included from
documentation integration tests so we detect compile and runtime issues earlier.

Closes #28131
2018-01-16 12:05:03 +01:00
Yannick Welsch 196c7b80dc
Never return null from Strings.tokenizeToStringArray (#28224)
This method has a different contract than all the other methods in this class, returning null instead of an empty array when receiving a null input. While switching over some methods from delimitedListToStringArray to this method tokenizeToStringArray, this resulted in unexpected nulls in some places of our code.

Relates #28213
2018-01-16 09:58:58 +01:00
Yannick Welsch 0c4e2cbc19
Fallback to TransportMasterNodeAction for cluster health retries (#28195)
ClusterHealthAction does not use the regular retry logic, possibly causing StackOverflowErrors.

Relates #28169
2018-01-16 09:50:06 +01:00
Nhat Nguyen 6c297ad7c8 TEST: Update logging for testAckedIndexing
- Log the response of indexing requests
- Correct logging setting for discovery package
2018-01-15 18:14:04 -05:00
Nicholas Knize 5ed25f1e12 [GEO] Add WKT Support to GeoBoundingBoxQueryBuilder
Add WKT BBOX parsing support to GeoBoundingBoxQueryBuilder.
2018-01-15 13:30:51 -06:00
Adrien Grand 0a92e43f62
Avoid doing redundant work when checking for self references. (#26927)
Currently we test all maps, arrays or iterables. However, in the case that maps
contain sub maps for instance, we will test the sub maps again even though the
work has already been done for the top-level map.

Relates #26907
2018-01-15 18:36:32 +01:00
Adrien Grand a16f80a832
Fix casts in HotThreads. (#27578)
Even though an overflow would be very unlikely, it's better to use the longs
directly in the comparator.
2018-01-15 18:35:27 +01:00
Adrien Grand 77a7e2480b
Allow update of `eager_global_ordinals` on `_parent`. (#28014)
A bug introduced in #24407 currently prevents `eager_global_ordinals` from
being updated. This new approach should fix the issue while still allowing
mapping updates to not specify the `_parent` field if it doesn't need
updating, which was the goal of #24407.
2018-01-15 18:34:10 +01:00
Jim Ferenczi bd11e6c441
Fix NPE on composite aggregation with sub-aggregations that need scores (#28129)
The composite aggregation defers the collection of sub-aggregations to a second pass that visits documents only if they
appear in the top buckets. Though the scorer for sub-aggregations is not set on this second pass and generates an NPE if any sub-aggregation
tries to access the score. This change creates a scorer for the second pass and makes sure that sub-aggs can use it safely to check the score of
the collected documents.
2018-01-15 18:30:38 +01:00
Tim Brooks ee7eac8dc1
`MockTcpTransport` to connect asynchronously (#28203)
The method `initiateChannel` on `TcpTransport` is explicit in that
channels can be connect asynchronously. All production implementations
do connect asynchronously. Only the blocking `MockTcpTransport`
connects in a synchronous manner. This avoids testing some of the
blocking code in `TcpTransport` that waits on connections to complete.
Additionally, it requires a more extensive method signature than
required for other transports.

This commit modifies the `MockTcpTransport` to make these connections
asynchronously on a different thread. Additionally, it simplifies that
`initiateChannel` method signature.
2018-01-15 10:20:30 -07:00
Jim Ferenczi 190f1e1fb3
Fix synonym phrase query expansion for cross_fields parsing (#28045)
* Fix synonym phrase query expansion for cross_fields parsing

The `cross_fields` mode for query parser ignores phrase query generated by multi-word synonyms.
In such case only the first field of each analyzer group is kept. This change fixes this issue
by expanding the phrase query for each analyzer group to **all** fields using a disjunction max query.
2018-01-15 18:00:20 +01:00
Tim Brooks 3895add2ca
Introduce elasticsearch-core jar (#28191)
This is related to #27933. It introduces a jar named elasticsearch-core
in the lib directory. This commit moves the JarHell class from server to
elasticsearch-core. Additionally, PathUtils and some of Loggers are
moved as JarHell depends on them.
2018-01-15 09:59:01 -07:00
Jim Ferenczi 5973c2bf31 #28218: Update the Lucene version for 6.2.0 after backport 2018-01-15 17:27:51 +01:00
Jim Ferenczi be012b1326
upgrade to lucene 7.2.1 (#28218) 2018-01-15 16:47:46 +01:00
Colin Goodheart-Smithe 023d08ee91
Adds metadata to rewritten aggregations (#28185)
* Adds metadata to rewritten aggregations

Previous to this change, if any filters in the filters aggregation were rewritten, the rewritten version of the FiltersAggregationBuilder would not contain the metadata form the original. This is because `AbstractAggregationBuilder.getMetadata()` returns an empty map when not metadata is set.

Closes #28170

* Always set metadata when rewritten
2018-01-15 08:44:49 +00:00
Igor Motov aec0c0f9b6 Update version of TaskInfo header serialization after backport
Update the serialization version after backporting #27764 to 6.x.
2018-01-14 19:20:32 -05:00
Nhat Nguyen fbb840b5c8 TEST: Tightens file-based condition in peer-recovery
As a replica always keeps a safe commit and starts peer-recovery with
that commit; file-based recovery  only happens if new operations are
added to the primary and the required translog is not fully retained. In
the test, we tried to produce this condition by flushing a new commit in
order to trim all translog. However, if the new global checkpoint is not
persisted yet, we will keep two commits and not trim translog. This
commit tightens the file-based condition in the test by waiting for the
global checkpoint persisted properly on the new primary before flushing.

Close #28209
Relates #28181
2018-01-13 22:03:30 -05:00
Nhat Nguyen 9774ba35a1 Correct backport replica rollback to 6.2 (#28181)
The previous backport was not corect.

Relates #28181
2018-01-13 14:10:23 -05:00
Nhat Nguyen 0151c1565d Backport replica rollback to 6.2 (#28181)
Relates #28181
2018-01-13 11:44:13 -05:00
Nhat Nguyen e44e34f42a Rename deleteLocalTranslog to createNewTranslog
We introduced a new option `createNewTranslog` in #28181. However, we
named that parameter as deleteLocalTranslog in other places. This commit
makes sure to have a consistent naming in these places.

Relates #28181
2018-01-13 11:44:13 -05:00
Nhat Nguyen fafdb8d9e3 AwaitsFix #testRecoveryAfterPrimaryPromotion
Relates #28209
2018-01-13 11:44:13 -05:00
Nhat Nguyen 82722ebad3 TEST: init unassigned gcp in testAcquireIndexCommit
The global checkpoint should be assigned to unassigned rather than 0. If
a single document is indexed and the global checkpoint is initialized
with 0, the first commit is safe which the test does not suppose.

Relates #28038
2018-01-12 20:09:34 -05:00
Nhat Nguyen 095f31b80e
Replica start peer recovery with safe commit (#28181)
Today a replica starts a peer-recovery with the last commit. If the last
commit is not a safe commit, a replica will immediately fallback to the
file based sync which is more expensive than the sequence based
recovery. This commit modifies the peer-recovery in replica to start
with a safe commit. Moreover we can keep the existing translog on the
target if the recovery is sequence based recovery.

Relates #10708
2018-01-12 19:09:31 -05:00
Nhat Nguyen f2db2a02e2
Truncate tlog cli should assign global checkpoint (#28192)
We are targeting to always have a safe index once the recovery is done. 
This invariant does not hold if the translog is manually truncated by 
users because the truncate translog cli resets the global checkpoint to
unassigned. This commit assigns the global checkpoint to the max_seqno
of the last commit when truncating translog. We can only safely do it
because the truncate translog command will generate a new history uuid
for that shard. With a new history UUID, sequence-based recovery between
that shard and other old shards will be disabled.

Relates #28181
2018-01-12 19:06:04 -05:00
Jason Tedor 1e330f795f Merge branch 'master' into ccr
* master:
  Fix lock accounting in releasable lock
  Add ability to associate an ID with tasks  (#27764)
  [DOCS] Removed differencies between text and code (#27993)
  text fixes (#28136)
  Update getting-started.asciidoc (#28145)
  [Docs] Spelling fix in painless-getting-started.asciidoc (#28187)
  Fixed the cat.health REST test to accept 4ms, not just 4.0ms (#28186)
  Do not keep 5.x commits once having 6.x commits (#28188)
2018-01-12 16:18:39 -05:00
Jason Tedor a15ba75d93
Fix lock accounting in releasable lock
Releasble locks hold accounting on who holds the lock when assertions
are enabled. However, the underlying lock can be re-entrant yet we mark
the lock as not held by the current thread as soon as the releasable is
closed. For a re-entrant lock this is not right because the thread could
have entered the lock multiple times. Instead, we have to count how many
times the thread has entered the lock and only mark the lock as not held
by the current thread when the counter reaches zero.

Relates #28202
2018-01-12 16:17:30 -05:00
Igor Motov c75ac319a6
Add ability to associate an ID with tasks (#27764)
Adds support for capturing the X-Opaque-Id header from a REST request and storing it's value in the tasks that this request started. It works for all user-initiated tasks (not only search).

Closes #23250

Usage:
```
$ curl -H "X-Opaque-Id: imotov" -H "foo:bar" "localhost:9200/_tasks?pretty&group_by=parents"
{
  "tasks" : {
    "7qrTVbiDQKiZfubUP7DPkg:6998" : {
      "node" : "7qrTVbiDQKiZfubUP7DPkg",
      "id" : 6998,
      "type" : "transport",
      "action" : "cluster:monitor/tasks/lists",
      "start_time_in_millis" : 1513029940042,
      "running_time_in_nanos" : 266794,
      "cancellable" : false,
      "headers" : {
        "X-Opaque-Id" : "imotov"
      },
      "children" : [
        {
          "node" : "V-PuCjPhRp2ryuEsNw6V1g",
          "id" : 6088,
          "type" : "netty",
          "action" : "cluster:monitor/tasks/lists[n]",
          "start_time_in_millis" : 1513029940043,
          "running_time_in_nanos" : 67785,
          "cancellable" : false,
          "parent_task_id" : "7qrTVbiDQKiZfubUP7DPkg:6998",
          "headers" : {
            "X-Opaque-Id" : "imotov"
          }
        },
        {
          "node" : "7qrTVbiDQKiZfubUP7DPkg",
          "id" : 6999,
          "type" : "direct",
          "action" : "cluster:monitor/tasks/lists[n]",
          "start_time_in_millis" : 1513029940043,
          "running_time_in_nanos" : 98754,
          "cancellable" : false,
          "parent_task_id" : "7qrTVbiDQKiZfubUP7DPkg:6998",
          "headers" : {
            "X-Opaque-Id" : "imotov"
          }
        }
      ]
    }
  }
}
```
2018-01-12 15:34:17 -05:00
Nhat Nguyen 55a14230a7
Do not keep 5.x commits once having 6.x commits (#28188)
Currently we keep a 5.x index commit as a safe commit until we have a
6.x safe commit. During that time, if peer-recovery happens, a primary
will send a 5.x commit in file-based sync and the recovery will even
fail as the snapshotted commit does not have sequence number tags.

This commit updates the combined deletion policy to delete legacy
commits if there are 6.x commits.

Relates #27606
Relates #28038
2018-01-11 18:34:17 -05:00
Jason Tedor 8f3e9d60ee Merge branch 'master' into ccr
* master: (43 commits)
  Rename core module to server (#28180)
  upgraded jna from 4.4.0-1 to 4.5.1 (#28183)
  [TEST] Do not call RandomizedTest.scaledRandomIntBetween from multiple threads
  Primary send safe commit in file-based recovery (#28038)
  [Docs] Correct response json in rank-eval.asciidoc
  Add scroll parameter to _reindex API (#28041)
  Include all sentences smaller than fragment_size in the unified highlighter (#28132)
  Modifies the JavaAPI docs related to AggregationBuilder
  [Docs] Improvements in script-fields.asciidoc (#28174)
  [Docs] Remove Kerberos/SPNEGO Shield plugin (#28019)
  Ignore null value for range field (#27845) (#28116)
  Fix environment variable substitutions in list setting (#28106)
  docs: Replaces indexed script java api docs with stored script api docs
  test: ensure we endup with a single segment
  Make sure that we don't detect files as maven coordinate when installing a plugin (#28163)
  [Tests] temporary disable meta plugin rest tests #28163
  meta-plugin should install bin and config at the top level (#28162)
  Painless: Add public member read/write access test. (#28156)
  Docs: Clarify password protection support with keystore (#28157)
  [Docs] fix plugin properties inclusion for plugins authors
  ...
2018-01-11 16:41:47 -05:00
Tim Brooks 99f88f15c5
Rename core module to server (#28180)
This is related to #27933. It renames the core module to server. This is
the first step towards introducing an elasticsearch-core jar.
2018-01-11 11:30:43 -07:00