Commit Graph

30414 Commits

Author SHA1 Message Date
Jack Conradson 5c1d3aa2f0
Painless: Fixes a null pointer exception in certain cases of for loop usage (#28506)
The initializer and afterthought were not having their types
appropriately cast which is necessary with expressions which in turn
caused values to be popped off the stack that were null.
2018-02-05 11:57:21 -08:00
Nik Everett 5003ef18ac
Scripts: Fix security for deprecation warning (#28485)
If you call `getDates()` on a long or date type field add a deprecation
warning to the response and log something to the deprecation logger.
This *mostly* worked just fine but if the deprecation logger happens to
roll then the roll will be performed with the script's permissions
rather than the permissions of the server. And scripts don't have
permissions to, say, open files. So the rolling failed. This fixes that
by wrapping the call the deprecation logger in `doPriviledged`.

This is a strange `doPrivileged` call because it doens't check
Elasticsearch's `SpecialPermission`. `SpecialPermission` is a permission
that no-script code has and that scripts never have. Usually all
`doPrivileged` calls check `SpecialPermission` to make sure that they
are not accidentally acting on behalf of a script. But in this case we
are *intentionally* acting on behalf of a script.

Closes #28408
2018-02-03 14:56:08 -05:00
Nhat Nguyen de6d31ebc2 Backport fail shard w/o marking as stale PR to v6.3
Relates #28054
2018-02-03 12:07:39 -05:00
Nhat Nguyen 965efa51cc
Allows failing shards without marking as stale (#28054)
Currently when failing a shard we also mark it as stale (eg. remove its
allocationId from from the InSync set). However in some cases, we need 
to be able to fail shards but keep them InSync set. This commit adds
such capacity. This is a preparatory change to make the primary-replica
resync less lenient.

Relates #24841
2018-02-03 09:41:53 -05:00
Andy Bristol 13083e27da
[TEST] packaging tests: clean up Vagrantfile (#28173)
* Consolidates provision steps so it's more clear which steps are
applied to all boxes
* Removes duplicate configuration that was being stomped
* Ensure rsync, a dependency for platform steps, is installed on linux
* Ruby style changes

For #26741
2018-02-02 12:04:45 -08:00
Nhat Nguyen 875bbfe699 Backported the harden synced-flush PR to v6.3.0
Relates #28464
2018-02-02 14:31:37 -05:00
Deb Adair 459233d550 [DOCS] Fixed list formatting. 2018-02-02 11:01:40 -08:00
Ryan Ernst a139c42964
Build: Fix test task to explicitly depend on testClasses task (#28490)
Gradle 4.5 now hides immutable task dependencies. We previously copied
the existing dependencies from the builtin test task to the
randomizedtesting task. This commit adds testClasses as an extra
dependency of the randomizedtesting task, to ensure the classes are
built.
2018-02-02 10:57:01 -08:00
Jack Conradson 90c74a7e09
Remove RuntimeClass from Painless Definition in favor of just Painless Struct. (#28486) 2018-02-02 10:26:02 -08:00
David Turner ab8f5ea54c
Forbid trappy methods from java.time (#28476)
ava.time has the functionality needed to deal with timezones with varying 
offsets correctly, but it also has a bunch of methods that silently let you
forget about the hard cases, which raises the risk that we'll quietly do the
wrong thing at some point in the future.

This change adds the trappy methods to the list of forbidden methods to try and
help stop this from happening.

It also fixes the only use of these methods in the codebase so far:
IngestDocument#deepCopy() used ZonedDateTime.of() which may alter the offset of
the given time in cases where the offset is ambiguous.
2018-02-02 18:24:02 +00:00
Lee Hinman 3ddea8d8d2
Start switching to non-deprecated ParseField.match method (#28488)
This commit switches all the modules and server test code to use the
non-deprecated `ParseField.match` method, passing in the parser's deprecation
handler or the logging deprecation handler when a parser is not available (like
in tests).

Relates to #28449
2018-02-02 10:10:13 -07:00
Nhat Nguyen 5f2121960e
Synced-flush should not seal index of out of sync replicas (#28464)
Today the correctness of synced-flush is guaranteed by ensuring that 
there is no ongoing indexing operations on the primary. Unfortunately, a
replica might fall out of sync with the primary even the condition is
met. Moreover, if synced-flush mistakenly issues a sync_id for an out of
sync replica, then that replica would not be able to recover from the 
primary. ES prevents that peer-recovery because it detects that both
indexes from primary and replica were sealed with the same sync_id but
have a different content. This commit modifies the synced-flush to not
issue sync_id for out of sync replicas. This change will report the
divergence issue earlier to users and also prevent replicas from getting
into the "unrecoverable" state.

Relates #10032
2018-02-02 11:20:38 -05:00
Luca Cavanna 075fdc579f
[DOCS] Update APIs grouping and ordering in REST high-level Client docs (#28497)
With this new structure supported APIs are ordered and grouped in the same way as they are in the Elasticsearch reference docs
2018-02-02 17:19:50 +01:00
Jim Ferenczi 7c2bcf3953
Mark synonym_graph as beta in the docs (#28496)
We do want to keep this functionality in the future and we provide support for it.
This change is a first step towards replacing the `synonym` token filter with `synonym_graph`.
2018-02-02 16:33:48 +01:00
Jim Ferenczi 88f4c1c03a
Add a test for sub-aggregations rewrite (#28491)
This commit adds a test to check that the rewrite of a sub-aggregation triggers a copy of the parent aggregation.

Relates #28430
Closes #27782
2018-02-02 16:04:33 +01:00
javanna b5986c8dce fix checkstyle error in SearchDocumentationIT 2018-02-02 14:05:30 +01:00
Clinton Gormley 45c1e37740 Add defined ID to terms agg size header 2018-02-02 13:43:20 +01:00
javanna 9d11cfd652 [DOCS] Remove rawtypes suppressions and fix violations in REST high-level client docs tests 2018-02-02 11:57:55 +01:00
javanna 174e243f83 [DOCS] Adapt indices exists docs after recent listener changes 2018-02-02 11:46:05 +01:00
Haris Osmanagić 897ef458f3 Add support for indices exists to REST high level client (#27384)
Relates to #27205
2018-02-02 11:25:36 +01:00
Yannick Welsch 031415a5f6
Replicate writes only to fully initialized shards (#28049)
The primary currently replicates writes to all other shard copies as soon as they're added to the routing table. Initially those shards are not even ready yet to receive these replication requests, for example when undergoing a file-based peer recovery. Based on the specific stage that the shard copies are in, they will throw different kinds of exceptions when they receive the replication requests. The primary then ignores responses from shards that match certain exception types. With this mechanism it's not possible for a primary to distinguish between a situation where a replication target shard is not allocated and ready yet to receive requests and a situation where the shard was successfully allocated and active but subsequently failed.
This commit changes replication so that only initializing shards that have successfully opened their engine are used as replication targets. This removes the need to replicate requests to initializing shards that are not even ready yet to receive those requests. This saves on network bandwidth and enables features that rely on the distinction between a "not-yet-ready" shard and a failed shard.
2018-02-02 11:13:07 +01:00
Christoph Büscher bc10334f7a
[Docs] Move callouts in range.asciidoc (#28264)
Currently the callouts for this section are below all the examples, making it
harder to relate them to the snippets. Instead they should be moved closer 
to the examples.
2018-02-02 11:00:07 +01:00
Luca Cavanna d10dec3e99
[DOCS] expand examples on providing mappings for create index and put mapping (#28483)
* [DOCS] expand examples on providing mappings for create index and put mapping

The create index API and put mappings API docs the for high-level Java REST client didn't have a lot of info on how to provide mappings. This commit adds some examples.
2018-02-02 10:32:24 +01:00
Jim Ferenczi c4e0a84344
Mark the composite aggregation as a beta feature (#28431)
The `composite` aggregation should be marked as beta (rather than experimental) in the documentation.
2018-02-02 09:24:10 +01:00
Nhat Nguyen 5be478f938 Remove uncommitted ops assertion in shouldFlush
This assertion does not hold if engine is flushed between the invocation
of translog.uncommittedSizeInBytes and translog.uncommittedOperations.
These two values can be calculated from different commits.
2018-02-01 18:20:56 -05:00
Lisa Cawley 3035f0b2f3
[DOCS] Build Elasticsearch Reference from elasticsearch repo (#28469) 2018-02-01 13:36:06 -08:00
Arne Neumann 3b41b98ee4 documentation fix: index exists is ambiguous (#28460) 2018-02-01 13:22:32 -08:00
Jack Conradson df1c696e1d
Remove Painless Type From Locals, Variables, Params, and ScriptInfo (#28471) 2018-02-01 12:46:20 -08:00
markharwood 998461c737
Test fix - reenable BWC tests and lower version checks now that PR 28440 for allowPartialSearchResults flag backported to 6.x (#28482)
Support for allowPartialSearchResults is now in 6.3 so changing master BWC checks accordingly
2018-02-01 19:20:21 +00:00
Nhat Nguyen 1970e01782
Add lower bound for translog flush threshold (#28382)
If the translog flush threshold is too small (eg. smaller than the
translog header), we may repeatedly flush even there is no uncommitted
operation because the shouldFlush condition can still be true after
flushing. This is currently avoided by adding an extra guard against the
uncommitted operations. However, this extra guard makes the shouldFlush
complicated. This commit replaces that extra guard by a lower bound for
translog flush threshold. We keep the lower bound small for convenience
in testing.

Relates #28350
Relates #23606
2018-02-01 13:51:53 -05:00
Tanguy Leroux bb97c00556
[Docs] Docs tests should wait for async execution to complete (#28481)
This commit splits the async execution documentation into 2 parts, one
for the async method itself and one for the action listener. This allows
to add more doc and to use CountDownLatches in doc tests to wait for
asynchronous operations to be completed before moving to the next test.

It also renames few files.

Related to #28457
2018-02-01 17:56:13 +01:00
Luca Cavanna d860971572
REST high-level client: add support for split and shrink index API (#28425)
Relates to #27205
2018-02-01 16:37:01 +01:00
Colin Goodheart-Smithe d37c59d9dd
Fixes Eclipse build after secure-sm project was added (#28480) 2018-02-01 15:33:26 +00:00
Tanguy Leroux ec187caeae
[Test] Fix CRUDDocumentationIT (#28457)
Similarly to other documentation tests in the high level client, the
asynchronous operation like update, index or delete can make the test
fail if it sneak in between the middle of another operation.

This commit moves the async doc tests to be the last ones executed and
adds assert busy loops to ensure that the asynchronous operations are
correctly terminated.

closes #28446
2018-02-01 16:30:17 +01:00
Martijn van Groningen 61806802fb
Add persistent tasks
Persistent tasks are build on top of node tasks and provide functionality to restart a task to run on a different coordination node in case the coordinating node is no longer available.
It is up to a persistent task implementation to keep track of status, so that in case the task is restarted, the task can continue were it left off before it was restarted.
2018-02-01 15:26:17 +01:00
Alan Woodward 9500513ba1
Move leftover aliases test from core/ to server/ (#28463) 2018-02-01 11:40:29 +00:00
olcbean 344db6c0b1 Java high-level REST client : minor fixes (#28467)
Corrected method name and a parameter name
2018-02-01 11:50:43 +01:00
Martijn van Groningen ecb1d07d00
percolator: remove deprecated map_unmapped_fields_as_string setting 2018-02-01 11:11:22 +01:00
Colin Goodheart-Smithe 65157e9428
[TEST] Replaces flaky breaker IT test with unit test (#28418)
This change remove the `CircuitBreakerIT. testParentChecking` test method which fails intermittently in unexpected ways with a `MemoryCircuitBreakerTests. testBorrowingSiblingBreakerMemory` unit test method which can test the borrowing functionality more directly

Closes #28223
2018-02-01 08:32:46 +00:00
Jim Ferenczi dd40b984c4
Add a shallow copy method to aggregation builders (#28430)
This change adds a shallow copy method for aggregation builders. This method returns a copy of the builder replacing the factoriesBuilder and metaDada
This method is used when the builder is rewritten (AggregationBuilder#rewrite) in order to make sure that we create a new instance of the parent builder when sub aggregations are rewritten.

Relates #27782
2018-02-01 09:22:32 +01:00
Jim Ferenczi c7d5a54b42
Fix AIOOB on indexed geo_shape query (#28458)
This change fixes a possible AIOOB during the parsing of the document that contains the indexed shape.
This change ensures that the parsing does not continue when the field that contains the shape has been found.

Closes #28456
2018-02-01 09:01:48 +01:00
Martijn van Groningen 9bada306dc
Improved percolator candidate query tests. 2018-02-01 07:43:03 +01:00
Jason Tedor 33009e8f7c
Bump Gradle to version 4.5
This commit bumps Gradle from version 4.3 to version 4.5.

Relates #28472
2018-01-31 23:18:28 -05:00
Jack Conradson e281d57d82
Remove Painless Type From Painless Method/Field (#28466) 2018-01-31 16:27:52 -08:00
Jason Tedor f792606f1c Re-disable BWC tests after inadvertent re-enabling
These tests were disabled to facilitate backport of a PR which is not
yet complete. These tests were accidentally reenabled after a merge
conflict was resolved in the wrong direction. This commit addresses this
issue.
2018-01-31 19:14:31 -05:00
Jason Tedor 2570dc5997 Remove leftover securesm files
These files were leftover after integrating securesm into
libs/secure-sm. This commit removes this leftover directory.
2018-01-31 18:49:22 -05:00
Jason Tedor 1b3d529bef Introduce secure security manager to project
This commit migrates SecureSM, our secure security manager
implementation, from its own repository to being a sub-project of
Elasticsearch.
2018-01-31 18:23:28 -05:00
Jason Tedor e6bd34e22c Tighten which classes can exit
Today a SecureSM security manager allows defining a list of packages
that can exit the VM. However, today there are no restrictions on
defining a package inside another JAR. This commit strengthens the
ability to prevent exit by allowing construction of SecureSM to be done
with a list of regular expressions (instead of a list of prefix names)
that classes will be tested against. With this, a security manager can
be installed that permits only exiting from an exact list of classes.

Relates #5
2018-01-31 18:14:52 -05:00
Jason Tedor 09057f24b5 Allow whitelist of packages that can exit
Today, SecureSM has a mechanism that enables a hardcoded list of test
packages to exit if the SecureSM instance is constructed with a boolean
flag indicating that these packages will be permitted to exit. This
commit replaces this mechanism by allowing the SecureSM instance to be
constructed with a whitelist of packages that can exit.

Relates #4
2018-01-31 18:14:52 -05:00
Jason Tedor 083f60e1da Add sanity interruption assertion
This commit adds an assertion to the TestSecureSM#testNoModifySibling
test that sanity checks that the second child was actually interrupted
and therefore actually attempted to interrupt the first child.
2018-01-31 18:14:52 -05:00