Commit Graph

29634 Commits

Author SHA1 Message Date
Kanako Nakai 23f85fe6d4 Fix max number of threads bootstrap docs
Previously the bootstrap check for max number of threads was increased
from 2048 to 4096 yet the docs were never adjusted for this change. This
commit addresses this so the docs are in-line with the limit enforced in
the bootstrap check.

Relates #27511
2017-11-28 22:19:04 -05:00
Yu 57da9dc57f Clarify setting IntelliJ preferences
This commit clarifies that the preferences menu in IntelliJ can differ
by OS (IntelliJ -> Preferences on macOS and Settings on Linux/Windows).

Relates #27575
2017-11-28 19:46:24 -05:00
Jack Conradson 2d927fabab
Painless: Fix errors allowing void to be assigned to def. (#27460) 2017-11-28 13:44:52 -08:00
Jack Conradson 9e42b77f7e
Painless: Fix variable scoping issue in lambdas not including captured variables. (#27571) 2017-11-28 13:30:13 -08:00
Simon Willnauer 4aa840698f
Ensure threadcontext is preserved when refresh listeners are invoked (#27565)
today a refresh listener won't preserve the entire context ie. won't carry
on response headers etc. from the caller side. This change adds support for
stored contexts.
2017-11-28 21:32:16 +01:00
Simon Willnauer 184b7f06ee
Make Segment statistics aware of segments hold by internal readers (#27558)
Today we only expose the external readers segments. Yet, from a statistics
perspective both internal and external segments are relevant. This commit
exposes the additional segments of the internal and external reader respectively.
2017-11-28 17:37:03 +01:00
Jason Tedor cefb46d0fc
Throw UOE from compressible bytes stream reset
A compressible bytes output stream is a stream output which supports a
reset method. However, compressible bytes output streams are unusual in
that the current implementation sometimes supports a reset (if the
stream is not compressed) and sometimes does not support a rest (if the
stream is compressed). This inconsistent behavior is puzzling and
instead we should simply always throw an unsupported operation
exception.

Relates #27564
2017-11-28 11:29:47 -05:00
Jim Ferenczi 37653c9dca [TEST] AggregationsIntegrationIT#testScroll can timeout
This change sets the scroll timeout for this test to 1m instead of 500ms in order
to avoid loosing the scroll on slow machines.

Relates #26378
2017-11-28 16:18:54 +01:00
Adrien Grand d01fcee645
Fix illegal cast of the "low cardinality" optimization of the `terms` aggregation. (#27543)
The GlobalOrdinalsStringTermsAggregator.LowCardinality aggregator casts global
values to `GlobalOrdinalMapping`, even though the implementation of global
values is different when a `missing` value is configured.

This commit adds a new API that gives access to the ordinal remapping in order
to fix this problem.
2017-11-28 14:55:09 +01:00
Adrien Grand 996990ad1f
Upgrade to lucene-7.2.0-snapshot-8c94404. (#27496)
The main highlight of this new snapshot is that it introduces the opportunity
for queries to opt out of caching. In case a query opts out of caching, not only
will it never be cached, but also no compound query that wraps it will be
cached.
2017-11-28 14:52:42 +01:00
Martijn van Groningen cb1204774b
Include the _index, _type and _id to nested search hits in the top_hits and inner_hits response.
Also include _type and _id for parent/child hits inside inner hits.

In the case of top_hits aggregation the nested search hits are
directly returned and are not grouped by a root or parent document, so
it is important to include the _id and _index attributes in order to know
to what documents these nested search hits belong to.

Closes #27053
2017-11-28 14:05:29 +01:00
David Turner a165d1df40
Minor improvements to docs for numeric types (#27553)
* Caps
* Fix awkward wording that took multiple passes to parse
* Floating point _number_
* Something more descriptive about the `scaled_float` scaling factor.
2017-11-28 11:36:07 +00:00
David Turner 7ac361d86e Update @AwaitsFix URL to point at an issue in the current repo 2017-11-28 09:35:46 +00:00
Nhat Nguyen 000f62c1d2
TEST: makes sure to corrupt referenced tlog files (#27546)
Method TruncateTranslogIT#corruptTranslogFiles corrupts some random
existing *.tlog files in a translog directory. However, this may not
actually corrupt translog at all if it corrupts only tlog files which
are not referenced by the Checkpoint (eg. their translog generations are
smaller the Checkpoint).

This commit makes sure that we corrupt some tlog files which are
referenced by the Checkpoint.

Closes #27538
2017-11-27 20:18:58 -05:00
Simon Willnauer 0eb87e5d57 [TEST] Fix broken test that still tried to acquire the shards to set it non-idle 2017-11-27 22:52:34 +01:00
Jason Tedor d8c28044da
Forbid granting the all permission in production
Running with the all permission java.security.AllPermission granted is
equivalent to disabling the security manager. This commit adds a
bootstrap check that forbids running with this permission granted.

Relates #27548
2017-11-27 16:05:27 -05:00
Jason Tedor 379d51fcfa
Bubble exceptions when closing compressible streams
Compressible bytes output stream swallows exceptions that occur when
closing. This commit changes this behavior so that such exceptions
bubble up.

Relates #27542
2017-11-27 13:48:04 -05:00
Simon Willnauer f23ed6188d
Skip shard refreshes if shard is `search idle` (#27500)
Today we refresh automatically in the background by default very second.
This default behavior has a significant impact on indexing performance
if the refreshes are not needed.
This change introduces a notion of a shard being `search idle` which a
shard transitions to after (default) `30s` without any access to an
external searcher. Once a shard is search idle all scheduled refreshes
will be skipped unless there are any refresh listeners registered.
If a search happens on a `serach idle` shard the search request _park_
on a refresh listener and will be executed once the next scheduled refresh
occurs. This will also turn the shard into the `non-idle` state immediately.

This behavior is only applied if there is no explicit refresh interval set.
2017-11-27 18:16:10 +01:00
Nhat Nguyen 8d6bfe53bb
Remove workaround in translog rest test (#27530)
Relates #25623 and a6db0ea908
2017-11-27 09:41:30 -05:00
Martijn van Groningen 3f98b85489
inner_hits: Return an empty _source for nested inner hit when filtering on a field that doesn't exist.
Before this change the search request would fail with an error indicating that it couldn't detect xcontent type based on the string: `null`
2017-11-27 10:51:24 +01:00
Christoph Büscher 1352b7c6ea
Use msearch instead of single search (#27520)
Change TransportRankEvalAction to use one MultiSearchRequest instead of issuing several parallel search requests to simplify the transport action.
2017-11-27 10:15:59 +01:00
Martijn van Groningen 4ab638b71d
percolator: Avoid TooManyClauses exception if number of terms / ranges is exactly equal to 1024
The logic whether to use CoveringQuery was in two places which is why this bug snug in.
2017-11-27 08:55:11 +01:00
Nhat Nguyen a4b4e14186
Dedup translog operations by reading in reverse (#27268)
Currently, translog operations are read and processed one by one. This
may be a problem as stale operations in translogs may suddenly reappear
in recoveries. To make sure that stale operations won't be processed, we
read the translog files in a reverse order (eg. from the most recent
file to the oldest file) and only process an operation if its sequence
number was not seen before.

Relates to #10708
2017-11-26 16:44:30 -05:00
Jason Tedor 0519fa223c
Ensure logging is configured for CLI commands
Any CLI commands that depend on core Elasticsearch might touch classes
(directly or indirectly) that depends on logging. If they do this and
logging is not configured, Log4j will dump status error messages to the
console. As such, we need to ensure that any such CLI command configures
logging (with a trivial configuration that dumps log messages to the
console). Previously we did this in the base CLI command but with the
refactoring of this class out of core Elasticsearch, we no longer
configure logging there (since we did not want this class to depend on
settings and logging). However, this meant for some CLI commands (like
the plugin CLI) we were no longer configuring logging. This commit adds
base classes between the low-level command and multi-command classes
that ensure that logging is configured. Any CLI command that depends on
core Elasticsearch should use this infrastructure to ensure logging is
configured. There is one exception to this: Elasticsearch itself because
it takes reponsibility into its own hands for configuring logging from
Elasticsearch settings and log4j2.properties. We preserve this special
status.

Relates #27523
2017-11-25 11:40:08 -05:00
Simon Willnauer a29dc20c26
Ensure `doc_stats` are changing even if refresh is disabled (#27505)
Today if refresh is disabled the doc stats are not updated anymore.
In a bulk index scenario this might cause confusion since even if
we refresh internal readers etc. doc stats are never advancing.
This change cuts over to the internal reader that is refreshed outside
of the external readers refresh interval but always equally `fresh` or
`fresher` which will cause less confusion.
2017-11-25 14:24:16 +01:00
Jason Tedor 0b6448726c
Fix classes that can exit
In a previous change, we locked down the classes that can exit by
specifying explicit classes rather than packages than can exit. Alas,
there was a bug in the sense that the class that we exit from in the
case of an uncaught exception is not
ElasticsearchUncaughtExceptionHandler but rather an anonymous nested
class of ElasticsearchUncaughtExceptionHandler. To address this, we
replace this anonymous class with a bonafide nested class
ElasticsearchUncaughtExceptionHandler$PrivilegedHaltAction. Note that if
we try to get this class name we have a $ in the middle of the string
which is a special regular expression character; as such, we have to
escape it.

Relates #27518
2017-11-24 19:00:18 -05:00
Nhat Nguyen e0e1a92d36 Revert "Adjust CombinedDeletionPolicy for multiple commits (#27456)"
The commit looks harmless, unfortunately it can break the engine flush
scheduler and the translog rolling. Both `uncommittedOperations` and
`uncommittedSizeInBytes` are currently calculated based on the minimum
required generation for recovery rather than the translog generation of
the last index commit. This is not correct if other index commits are
reserved for snapshotting even though we are keeping the last index
commit only.

This reverts commit e95d18ec23.
2017-11-24 15:19:50 -05:00
David Turner 00867e618d
Transpose expected and actual, and remove duplicate info from message. (#27515)
Previously:
```
   > Throwable #1: java.lang.AssertionError: Expected all shards successful but got successful [8] total [9]
   > Expected: <8>
   >      but: was <9>
```

Now:
```
   > Throwable #1: java.lang.AssertionError: Expected all shards successful
   > Expected: <9>
   >      but: was <8>
```
2017-11-24 17:45:34 +00:00
lcawley af971b3081 [DOCS] Fixed broken link in breaking changes 2017-11-24 09:16:14 -08:00
Christoph Büscher 5661b1c3df Merge branch 'master' into rankeval 2017-11-24 16:25:05 +01:00
Nhat Nguyen 06d35f4f01 Backport wait_for_initialiazing_shards to cluster health API
Relates #27489
2017-11-24 09:56:16 -05:00
Simon Willnauer 17e9940fc1
Carry over version map size to prevent excessive resizing (#27516)
Today we create a new concurrent hash map everytime we refresh
the internal reader. Under defaults this isn't much of a deal but
once the refresh interval is set to `-1` these maps grow quite large
and it can have a significant impact on indexing throughput. Under low
memory situations this can cause up to 2x slowdown. This change carries
over the map size as the initial capacity wich will be auto-adjusted once
indexing stops.

Closes #20498
2017-11-24 14:57:31 +01:00
Jim Ferenczi c6724abe74
Fix scroll query with a sort that is a prefix of the index sort (#27498)
During a scroll, if the search sort matches the index sort we use the sort values of the last doc returned by
the previous scroll to optimize the main query with a `SearchAfterSortedDocQuery`.
This query can "jump" directly to the first document that sorts after the provided sort values.
This optim is also applied if the search sort is a prefix of the index sort but this case throws an exception
because we use the index sort (instead of the search sort) to validate the sort values of the last document.
This change fixes this bug and adds a test for it.
2017-11-24 13:44:47 +01:00
Tanguy Leroux 5dc5580eac
Delete shard store files before restoring a snapshot (#27476)
Pull request #20220 added a change where the store files
that have the same name but are different from the ones in the
snapshot are deleted first before the snapshot is restored.
This logic was based on the `Store.RecoveryDiff.different`
set of files which works by computing a diff between an
existing store and a snapshot.

This works well when the files on the filesystem form valid
shard store, ie there's a `segments` file and store files
are not corrupted. Otherwise, the existing store's snapshot
metadata cannot be read (using Store#snapshotStoreMetadata())
and an exception is thrown
(CorruptIndexException, IndexFormatTooOldException etc) which
is later caught as the begining of the restore process
(see RestoreContext#restore()) and is translated into
an empty store metadata (Store.MetadataSnapshot.EMPTY).

This will make the deletion of different files introduced
in #20220 useless as the set of files will always be empty
even when store files exist on the filesystem. And if some
files are present within the store directory, then restoring
a snapshot with files with same names will fail with a
FileAlreadyExistException.

This is part of the #26865 issue.

There are various cases were some files could exist in the
 store directory before a snapshot is restored. One that
Igor identified is a restore attempt that failed on a node
and only first files were restored, then the shard is allocated
again to the same node and the restore starts again (but fails
 because of existing files). Another one is when some files
of a closed index are corrupted / deleted and the index is
restored.

This commit adds a test that uses the infrastructure provided
by IndexShardTestCase in order to test that restoring a shard
succeed even when files with same names exist on filesystem.

Related to #26865
2017-11-24 13:15:34 +01:00
kel 4885acb048 Replace `delimited_payload_filter` by `delimited_payload` (#26625)
The `delimited_payload_filter` is renamed to `delimited_payload`, the old name is 
deprecated and should be replaced by `delimited_payload`.

Closes #21978
2017-11-24 13:03:19 +01:00
David Turner e8c2e5d6dd
CURRENT should not be a -SNAPSHOT version if build.snapshot is false (#27512) 2017-11-24 11:19:41 +00:00
Alexander Kazakov 43a91f4184 Fix merging of _meta field (#27352) 2017-11-24 09:44:39 +01:00
David Turner 8916935639
Remove unused method (#27508) 2017-11-24 08:09:31 +00:00
Martijn van Groningen f1ebf366bf
unmuted test, this has been fixed by #27397
Closes #27497
2017-11-24 08:53:00 +01:00
David Turner 89ba8996c6 Consolidate version numbering semantics (#27397)
Fixes to the build system, particularly around BWC testing, and to make future
version bumps less painful.
2017-11-23 20:21:53 +00:00
Nhat Nguyen 46b508d6c9
Add wait_for_no_initializing_shards to cluster health API (#27489)
This adds a new option to the cluster health request allowing to wait
until there is no initializing shards.

Closes #25623
2017-11-23 15:09:58 -05:00
Simon Willnauer 93a988c557 [TEST] use routing partition size based on the max routing shards of the second split 2017-11-23 17:59:07 +01:00
Nhat Nguyen e95d18ec23
Adjust CombinedDeletionPolicy for multiple commits (#27456)
Today, we keep only the last index commit and use only it to calculate
the minimum required translog generation. This may no longer be correct
as we introduced a new deletion policy which keeps multiple index
commits. This change adjusts the CombinedDeletionPolicy so that it can
work correctly with a new index deletion policy.

Relates to #10708, #27367
2017-11-23 11:34:50 -05:00
Clinton Gormley d1b1d711df Update composite-aggregation.asciidoc
Fixed asciidoc typo
2017-11-23 15:05:14 +01:00
olcbean fd564b10db Deprecate `levenstein` in favor of `levenshtein` (#27409)
Support both spellings thoughout 6.x, reporting the incorrect one as deprecated.
2017-11-23 12:53:47 +00:00
Christoph Büscher 5735477283 Fix some documentation typos 2017-11-23 12:31:25 +01:00
Simon Willnauer fadbe0de08
Automatically prepare indices for splitting (#27451)
Today we require users to prepare their indices for split operations.
Yet, we can do this automatically when an index is created which would
make the split feature a much more appealing option since it doesn't have
any 3rd party prerequisites anymore.

This change automatically sets the number of routinng shards such that
an index is guaranteed to be able to split once into twice as many shards.
The number of routing shards is scaled towards the default shard limit per index
such that indices with a smaller amount of shards can be split more often than
larger ones. For instance an index with 1 or 2 shards can be split 10x
(until it approaches 1024 shards) while an index created with 128 shards can only
be split 3x by a factor of 2. Please note this is just a default value and users
can still prepare their indices with `index.number_of_routing_shards` for custom
splitting.

NOTE: this change has an impact on the document distribution since we are changing
the hash space. Documents are still uniformly distributed across all shards but since
we are artificually changing the number of buckets in the consistent hashign space
document might be hashed into different shards compared to previous versions.

This is a 7.0 only change.
2017-11-23 09:48:54 +01:00
olcbean 05998f91d0 Validate `op_type` for `_create` (#27483) 2017-11-23 07:19:47 +01:00
Nicholas Knize 4017049bac Minor ShapeBuilder cleanup
Remove some auto generated comments and make CoordinateNode ctors package private.
2017-11-22 23:08:15 -06:00
Martijn van Groningen ca9c476d88
muted test 2017-11-22 19:18:35 +01:00