Commit Graph

29821 Commits

Author SHA1 Message Date
Christoph Büscher 409b3d2ebd Revert "[Docs] Fix base directory to include for put_mapping.asciidoc"
This reverts commit 4f5be7db3c.
2018-01-16 17:30:55 +01:00
Martijn van Groningen 853f7e8780
Added multi get api to the high level rest client.
Relates to #27205
2018-01-16 17:27:02 +01:00
Christoph Büscher d4ac0026fc
[Docs] Clarify numeric datatype ranges (#28240)
Since #25826 we reject infinite values for float, double and half_float
datatypes. This change adds this restriction to the documentation for the
supported datatypes.

Closes #27653
2018-01-16 15:53:28 +01:00
Christoph Büscher 4f5be7db3c [Docs] Fix base directory to include for put_mapping.asciidoc 2018-01-16 15:19:47 +01:00
Nhat Nguyen 65e90079ad
Open engine should keep only starting commit (#28228)
Keeping unsafe commits when opening an engine can be problematic because
these commits are not safe at the recovering time but they can suddenly
become safe in the future. The following issues can happen if unsafe
commits are kept oninit.

1. Replica can use unsafe commit in peer-recovery. This happens when a
replica with a safe commit c1 (max_seqno=1) and an unsafe commit c2
(max_seqno=2) recovers from a primary with c1(max_seqno=1). If a new
document (seqno=2) is added without flushing, the global checkpoint is
advanced to 2; and the replica recovers again, it will use the unsafe
commit c2 (max_seqno=2 <= gcp=2) as the starting commit for sequenced
based recovery even the commit c2 contains a stale operation and the
document (with seqno=2) will not be replicated to the replica.

2. Min translog gen for recovery can go backwards in peer-recovery. This
happens when a replica with a safe commit c1 (local_checkpoint=1,
recovery_translog_gen=1) and an unsafe commit c2 (local_checkpoint=2,
recovery_translog_gen=2). The replica recovers from a primary, and keeps
c2 as the last commit, then sets last_translog_gen to 2. Flushing a new
commit on the replica will cause exception as the new last commit c3
will have recovery_translog_gen=1. The recovery translog generation of a
commit is calculated based on the current local checkpoint. The local
checkpoint of c3 is 1 while the local checkpoint of c2 is 2.

3. Commit without translog can be used for recovery. An old index, which
was created before multiple-commits is introduced (v6.2), may not have a
safe commit. If that index has a snapshotted commit without translog and
an unsafe commit, the policy can consider the snapshotted commit as a
safe commit for recovery even the commit does not have translog.

These issues can be avoided if the combined deletion policy keeps only
the starting commit onInit.

Relates #27804
Relates #28181
2018-01-16 08:37:42 -05:00
Christoph Büscher 67c1f1c856
[Docs] Fix Java Api index administration usage (#28133)
The Java API documentation for index administration currenty is wrong because
the PutMappingRequestBuilder#setSource(Object... source) and
CreateIndexRequestBuilder#addMapping(String type, Object... source) methods
delegate to methods that check that the input arguments are valid key/value
pairs:

https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-admin-indices.html

This changes the docs so the java api code examples are included from
documentation integration tests so we detect compile and runtime issues earlier.

Closes #28131
2018-01-16 12:05:03 +01:00
Adrien Grand efe2e52118
Fix eclipse build. (#28236)
Relates #28191
2018-01-16 10:50:07 +01:00
Yannick Welsch 196c7b80dc
Never return null from Strings.tokenizeToStringArray (#28224)
This method has a different contract than all the other methods in this class, returning null instead of an empty array when receiving a null input. While switching over some methods from delimitedListToStringArray to this method tokenizeToStringArray, this resulted in unexpected nulls in some places of our code.

Relates #28213
2018-01-16 09:58:58 +01:00
Yannick Welsch 0c4e2cbc19
Fallback to TransportMasterNodeAction for cluster health retries (#28195)
ClusterHealthAction does not use the regular retry logic, possibly causing StackOverflowErrors.

Relates #28169
2018-01-16 09:50:06 +01:00
fbsolo 71ba314c73 [Docs] Changes to ingest.asciidoc (#28212) 2018-01-16 09:36:19 +01:00
Nhat Nguyen 6c297ad7c8 TEST: Update logging for testAckedIndexing
- Log the response of indexing requests
- Correct logging setting for discovery package
2018-01-15 18:14:04 -05:00
Nicholas Knize 5ed25f1e12 [GEO] Add WKT Support to GeoBoundingBoxQueryBuilder
Add WKT BBOX parsing support to GeoBoundingBoxQueryBuilder.
2018-01-15 13:30:51 -06:00
Ryan Ernst 18463e7e9f
Painless: Add whitelist extensions (#28161)
This commit adds a PainlessExtension which may be plugged in via SPI to
add additional classes, methods and members to the painless whitelist on
a per context basis. An example plugin adding and using a whitelist is
also added.
2018-01-15 11:28:31 -08:00
Jim Ferenczi b82017cbfe
Fix daitch_mokotoff phonetic filter to use the dedicated Lucene filter (#28225)
This commit changes the phonetic filter factory to use a DaitchMokotoffSoundexFilter
instead of a PhoneticFilter with a daitch_mokotoff encoder when daitch_mokotoff is selected.
The latter does not hanlde branching when computing the soundex and fails to encode multiple
variations when possible.

Closes #28211
2018-01-15 19:35:54 +01:00
Adrien Grand 0a92e43f62
Avoid doing redundant work when checking for self references. (#26927)
Currently we test all maps, arrays or iterables. However, in the case that maps
contain sub maps for instance, we will test the sub maps again even though the
work has already been done for the top-level map.

Relates #26907
2018-01-15 18:36:32 +01:00
Adrien Grand a16f80a832
Fix casts in HotThreads. (#27578)
Even though an overflow would be very unlikely, it's better to use the longs
directly in the comparator.
2018-01-15 18:35:27 +01:00
Adrien Grand 05e851f0b0
Ignore the `-snapshot` suffix when comparing the Lucene version in the build and the docs. (#27927)
Currently if the Lucene version is `X.Y.Z-snapshot-{gitrev}`, then we will
expect the docs to have `X.Y.Z-snapshot` as a Lucene version. I would like
to change it to `X.Y.Z` so that this doesn't need changing when we move from a
snapshot to a final release.
2018-01-15 18:34:38 +01:00
Adrien Grand 77a7e2480b
Allow update of `eager_global_ordinals` on `_parent`. (#28014)
A bug introduced in #24407 currently prevents `eager_global_ordinals` from
being updated. This new approach should fix the issue while still allowing
mapping updates to not specify the `_parent` field if it doesn't need
updating, which was the goal of #24407.
2018-01-15 18:34:10 +01:00
Jim Ferenczi bd11e6c441
Fix NPE on composite aggregation with sub-aggregations that need scores (#28129)
The composite aggregation defers the collection of sub-aggregations to a second pass that visits documents only if they
appear in the top buckets. Though the scorer for sub-aggregations is not set on this second pass and generates an NPE if any sub-aggregation
tries to access the score. This change creates a scorer for the second pass and makes sure that sub-aggs can use it safely to check the score of
the collected documents.
2018-01-15 18:30:38 +01:00
Tim Brooks ee7eac8dc1
`MockTcpTransport` to connect asynchronously (#28203)
The method `initiateChannel` on `TcpTransport` is explicit in that
channels can be connect asynchronously. All production implementations
do connect asynchronously. Only the blocking `MockTcpTransport`
connects in a synchronous manner. This avoids testing some of the
blocking code in `TcpTransport` that waits on connections to complete.
Additionally, it requires a more extensive method signature than
required for other transports.

This commit modifies the `MockTcpTransport` to make these connections
asynchronously on a different thread. Additionally, it simplifies that
`initiateChannel` method signature.
2018-01-15 10:20:30 -07:00
Jim Ferenczi 190f1e1fb3
Fix synonym phrase query expansion for cross_fields parsing (#28045)
* Fix synonym phrase query expansion for cross_fields parsing

The `cross_fields` mode for query parser ignores phrase query generated by multi-word synonyms.
In such case only the first field of each analyzer group is kept. This change fixes this issue
by expanding the phrase query for each analyzer group to **all** fields using a disjunction max query.
2018-01-15 18:00:20 +01:00
Tim Brooks 3895add2ca
Introduce elasticsearch-core jar (#28191)
This is related to #27933. It introduces a jar named elasticsearch-core
in the lib directory. This commit moves the JarHell class from server to
elasticsearch-core. Additionally, PathUtils and some of Loggers are
moved as JarHell depends on them.
2018-01-15 09:59:01 -07:00
Jim Ferenczi 5973c2bf31 #28218: Update the Lucene version for 6.2.0 after backport 2018-01-15 17:27:51 +01:00
Jim Ferenczi be012b1326
upgrade to lucene 7.2.1 (#28218) 2018-01-15 16:47:46 +01:00
hanbj 784eba86b2 [Docs] Fix an error in painless-types.asciidoc (#28221) 2018-01-15 15:10:22 +01:00
Colin Goodheart-Smithe 023d08ee91
Adds metadata to rewritten aggregations (#28185)
* Adds metadata to rewritten aggregations

Previous to this change, if any filters in the filters aggregation were rewritten, the rewritten version of the FiltersAggregationBuilder would not contain the metadata form the original. This is because `AbstractAggregationBuilder.getMetadata()` returns an empty map when not metadata is set.

Closes #28170

* Always set metadata when rewritten
2018-01-15 08:44:49 +00:00
Igor Motov aec0c0f9b6 Update version of TaskInfo header serialization after backport
Update the serialization version after backporting #27764 to 6.x.
2018-01-14 19:20:32 -05:00
Nhat Nguyen fbb840b5c8 TEST: Tightens file-based condition in peer-recovery
As a replica always keeps a safe commit and starts peer-recovery with
that commit; file-based recovery  only happens if new operations are
added to the primary and the required translog is not fully retained. In
the test, we tried to produce this condition by flushing a new commit in
order to trim all translog. However, if the new global checkpoint is not
persisted yet, we will keep two commits and not trim translog. This
commit tightens the file-based condition in the test by waiting for the
global checkpoint persisted properly on the new primary before flushing.

Close #28209
Relates #28181
2018-01-13 22:03:30 -05:00
Nhat Nguyen 9774ba35a1 Correct backport replica rollback to 6.2 (#28181)
The previous backport was not corect.

Relates #28181
2018-01-13 14:10:23 -05:00
Nhat Nguyen 0151c1565d Backport replica rollback to 6.2 (#28181)
Relates #28181
2018-01-13 11:44:13 -05:00
Nhat Nguyen e44e34f42a Rename deleteLocalTranslog to createNewTranslog
We introduced a new option `createNewTranslog` in #28181. However, we
named that parameter as deleteLocalTranslog in other places. This commit
makes sure to have a consistent naming in these places.

Relates #28181
2018-01-13 11:44:13 -05:00
Nhat Nguyen fafdb8d9e3 AwaitsFix #testRecoveryAfterPrimaryPromotion
Relates #28209
2018-01-13 11:44:13 -05:00
Nhat Nguyen 82722ebad3 TEST: init unassigned gcp in testAcquireIndexCommit
The global checkpoint should be assigned to unassigned rather than 0. If
a single document is indexed and the global checkpoint is initialized
with 0, the first commit is safe which the test does not suppose.

Relates #28038
2018-01-12 20:09:34 -05:00
Nhat Nguyen 095f31b80e
Replica start peer recovery with safe commit (#28181)
Today a replica starts a peer-recovery with the last commit. If the last
commit is not a safe commit, a replica will immediately fallback to the
file based sync which is more expensive than the sequence based
recovery. This commit modifies the peer-recovery in replica to start
with a safe commit. Moreover we can keep the existing translog on the
target if the recovery is sequence based recovery.

Relates #10708
2018-01-12 19:09:31 -05:00
Nhat Nguyen f2db2a02e2
Truncate tlog cli should assign global checkpoint (#28192)
We are targeting to always have a safe index once the recovery is done. 
This invariant does not hold if the translog is manually truncated by 
users because the truncate translog cli resets the global checkpoint to
unassigned. This commit assigns the global checkpoint to the max_seqno
of the last commit when truncating translog. We can only safely do it
because the truncate translog command will generate a new history uuid
for that shard. With a new history UUID, sequence-based recovery between
that shard and other old shards will be disabled.

Relates #28181
2018-01-12 19:06:04 -05:00
Jason Tedor a15ba75d93
Fix lock accounting in releasable lock
Releasble locks hold accounting on who holds the lock when assertions
are enabled. However, the underlying lock can be re-entrant yet we mark
the lock as not held by the current thread as soon as the releasable is
closed. For a re-entrant lock this is not right because the thread could
have entered the lock multiple times. Instead, we have to count how many
times the thread has entered the lock and only mark the lock as not held
by the current thread when the counter reaches zero.

Relates #28202
2018-01-12 16:17:30 -05:00
Igor Motov c75ac319a6
Add ability to associate an ID with tasks (#27764)
Adds support for capturing the X-Opaque-Id header from a REST request and storing it's value in the tasks that this request started. It works for all user-initiated tasks (not only search).

Closes #23250

Usage:
```
$ curl -H "X-Opaque-Id: imotov" -H "foo:bar" "localhost:9200/_tasks?pretty&group_by=parents"
{
  "tasks" : {
    "7qrTVbiDQKiZfubUP7DPkg:6998" : {
      "node" : "7qrTVbiDQKiZfubUP7DPkg",
      "id" : 6998,
      "type" : "transport",
      "action" : "cluster:monitor/tasks/lists",
      "start_time_in_millis" : 1513029940042,
      "running_time_in_nanos" : 266794,
      "cancellable" : false,
      "headers" : {
        "X-Opaque-Id" : "imotov"
      },
      "children" : [
        {
          "node" : "V-PuCjPhRp2ryuEsNw6V1g",
          "id" : 6088,
          "type" : "netty",
          "action" : "cluster:monitor/tasks/lists[n]",
          "start_time_in_millis" : 1513029940043,
          "running_time_in_nanos" : 67785,
          "cancellable" : false,
          "parent_task_id" : "7qrTVbiDQKiZfubUP7DPkg:6998",
          "headers" : {
            "X-Opaque-Id" : "imotov"
          }
        },
        {
          "node" : "7qrTVbiDQKiZfubUP7DPkg",
          "id" : 6999,
          "type" : "direct",
          "action" : "cluster:monitor/tasks/lists[n]",
          "start_time_in_millis" : 1513029940043,
          "running_time_in_nanos" : 98754,
          "cancellable" : false,
          "parent_task_id" : "7qrTVbiDQKiZfubUP7DPkg:6998",
          "headers" : {
            "X-Opaque-Id" : "imotov"
          }
        }
      ]
    }
  }
}
```
2018-01-12 15:34:17 -05:00
akadko 6a5807ad8f [DOCS] Removed differencies between text and code (#27993) 2018-01-12 10:36:48 -05:00
Andrew Banchich a10c406292 text fixes (#28136) 2018-01-12 10:21:39 -05:00
Boudewijn 3a96518b58 Update getting-started.asciidoc (#28145)
Replaced single quotation marks with double quotation marks surrounding localhost address
2018-01-12 10:21:16 -05:00
Andrew Banchich c417427ecd [Docs] Spelling fix in painless-getting-started.asciidoc (#28187) 2018-01-12 11:52:15 +01:00
Clinton Gormley f15c2044c4
Fixed the cat.health REST test to accept 4ms, not just 4.0ms (#28186) 2018-01-12 11:19:57 +01:00
Nhat Nguyen 55a14230a7
Do not keep 5.x commits once having 6.x commits (#28188)
Currently we keep a 5.x index commit as a safe commit until we have a
6.x safe commit. During that time, if peer-recovery happens, a primary
will send a 5.x commit in file-based sync and the recovery will even
fail as the snapshotted commit does not have sequence number tags.

This commit updates the combined deletion policy to delete legacy
commits if there are 6.x commits.

Relates #27606
Relates #28038
2018-01-11 18:34:17 -05:00
Tim Brooks 99f88f15c5
Rename core module to server (#28180)
This is related to #27933. It renames the core module to server. This is
the first step towards introducing an elasticsearch-core jar.
2018-01-11 11:30:43 -07:00
Martijn van Groningen 7d0eb3292b
upgraded jna from 4.4.0-1 to 4.5.1 (#28183)
Closes #28172
2018-01-11 17:31:10 +01:00
Yannick Welsch 52d627b66f [TEST] Do not call RandomizedTest.scaledRandomIntBetween from multiple threads
KeyedLockTests calls RandomizedTest.scaledRandomIntBetween from multiple threads which uses RandomizedTest.isNightly()
whereas that method is not concurrency-safe (see implementation of GroupEvaluator.isGroupEnabled)
2018-01-11 16:43:23 +01:00
Nhat Nguyen 626c3d1fda
Primary send safe commit in file-based recovery (#28038)
Today a primary shard transfers the most recent commit point to a 
replica shard in a file-based recovery. However, the most recent commit
may not be a "safe" commit; this causes a replica shard not having a
safe commit point until it can retain a safe commit by itself.

This commits collapses the snapshot deletion policy into the combined 
deletion policy and modifies the peer recovery source to send a safe
commit.

Relates #10708
2018-01-11 10:39:12 -05:00
Christoph Büscher 39ff7b5a3f [Docs] Correct response json in rank-eval.asciidoc 2018-01-11 15:52:11 +01:00
Yu 228f7ffcdf Add scroll parameter to _reindex API (#28041)
Be able to change scroll timeout in _reindex API (by default: 5m)
2018-01-11 14:40:04 +01:00
Jim Ferenczi 87c841d178
Include all sentences smaller than fragment_size in the unified highlighter (#28132)
The unified highlighter selects a single sentence per fragment from the offset of the first highlighted term.
This change modifies this selection and allows more than one sentence in a single fragment.
The expansion is done forward (on the right of the matching offset), sentences are added to the current fragment iff the overall size of the fragment is smaller than the maximum length (fragment_size).
We should also add a way to expand the left context with the surrounding sentences but this is currently avoided because the unified highlighter in Lucene uses only the first offset that matches the query to derive the start and end offset of the next fragment.
If we expand on the left we could split multiple terms that would be grouped otherwise. Breaking this limitation implies some changes in the core of the unified highlighter.

Closes #28089
2018-01-11 13:26:11 +01:00