Commit Graph

26306 Commits

Author SHA1 Message Date
javanna f4aab0138d introduce ToXContentObject interface
`ToXContentObject` extends `ToXContent` without adding new methods to it, while allowing to mark classes that output complete xcontent objects to distinguish them from classes that require starting and ending an anonymous object externally.

Ideally ToXContent would be renamed to ToXContentFragment, but that would be a huge change in our codebase, hence we simply document the fact that toXContent outputs fragments with no guarantees that the output is valid per se without an external ancestor.

Relates to #16347
2017-01-06 23:31:48 +01:00
animageofmine e3546d59c4 Add support for ca-central-1 region to EC2 and S3 plugins
Closes #22458 #22454
2017-01-06 16:27:08 -06:00
Ryan Ernst 5adb3f992c Merge pull request #22443 from rjernst/disable_assertions
Use tests.asserts flag to allow disabling assertions
2017-01-06 12:03:45 -08:00
Ryan Ernst d235e489e6 Merge pull request #22335 from rjernst/keystore
Settings: Add infrastructure for elasticsearch keystore
2017-01-06 11:48:17 -08:00
Ryan Ernst cd6e3f4cea Merge branch 'master' into keystore 2017-01-06 09:32:08 -08:00
Ryan Ernst 42ebfe7bdb fix NPE 2017-01-06 09:11:07 -08:00
Tim B b9c2c2f6f0 Move IfConfig.logIfNecessary call into bootstrap (#22455)
This is related to #22116. A logIfNecessary() call makes a call to
NetworkInterface.getInterfaceAddresses() requiring SocketPermission
connect privileges. By moving this to bootstrap the logging call can be
made before installing the SecurityManager.
2017-01-06 11:10:53 -06:00
Nik Everett f24ca5188a Fix some issues with painless's strings (#22393)
1. Escape sequences we're working. For example `\\` is now correctly
interpreted as `\` instead of `\\`. Same with `\'` being `'` and
`\"` being `"`.
2. `'` delimited strings weren't allowed to contain `"`s but it looked
like they were intended to support it. Now they do.
3. Improves the error message when the script contains an invalid
escape sequence inside a string to include a list of the valid
escape sequences.

Closes #22372
2017-01-06 11:35:22 -05:00
Ali Beyad a487b90498 [TEST] fix explain API rest test that assumes there is only a single
node in the cluster (incorrect assumption)
2017-01-06 11:07:40 -05:00
Yannick Welsch 0035f5ab95 Fix compilation of benchmarks on JDK 9
The JDK 9 compiler (b151) emits the warning "No processor claimed any of these annotations" for annotations that would be runtime annotation. Maybe a
regression from https://bugs.openjdk.java.net/browse/JDK-8039469. This is a quick fix so that compilation works again.
2017-01-06 16:50:43 +01:00
Simon Willnauer 79093e1663 Ensure shrunk indices carry over version information from its source (#22469)
Today when an index is shrunk the version information is not carried over
from the source to the target index. This can cause major issues like mapping
incompatibilities for instance if an index from a previous major version is shrunk.

This commit ensures that all version information from the soruce index is preserved
when a shrunk index is created.

Closes #22373
2017-01-06 16:36:43 +01:00
Nik Everett 923820c6c9 Document the `detailed` parameter of tasks API (#22425)
Provides an example of using is and an example return description
and explains that we've added descriptions for some tasks but not
even close to all of them. And that we expect to change the
descriptions as we learn more.

Closes #22407

* Fix example

Getting a single task is always detailed, no need to specify.

* Rewrite like imotov wants it
2017-01-06 10:24:52 -05:00
Jason Tedor 9219d667e8 Disable the Netty recycler and pooled allocator
Netty plays a lot of games with recycling byte buffers in thread local
caches, and using a pooled byte buffer allocator to reduce pressure on
the garbage collector.

The recycler in particular appears to be fraught with peril. It appears
that there are circumstances where the recycler does not recycle quickly
enough and can exceed its capacity leading to heap exhaustion and out of
memory errors. If you spend a few minutes reading the history of the
recycler on the Netty GitHub issues, it appears it has been nothing but
a source of trouble, and the project itself has an open issue that
proposes disabling by default and possibly even removing the recycler.

The pooled byte buffer allocator has problems itself. It sizes the pool
based on the number of runtime processors and can indeed grab a very
large percentage of the heap (in some cases 50% or more). Additionally,
the Netty project continues to struggle with leaks here.

We are seeing users struggle with issues in 5.x that I think are largely
driven by some of the problems here with Netty.

This change proposes to disable the recycler, and to disable the pooled
byte buffer allocator. I think that disabling these features will return
some of the stablity that these features appear to be losing us.

I have done performance testing on my workstation with disabling these
and I do not see a difference in performance. I propose that we make
this change in master and let some nightly benchmarks run to confirm
that there is not a difference in performance. If we are comfortable
with the performance changes, I propose backporting this to all active
branches.

Relates #22452
2017-01-06 06:37:49 -05:00
Simon Willnauer 56082d1028 add more tests for RemoteClusterService 2017-01-06 12:35:18 +01:00
Simon Willnauer 418ec62bfb Merge branch 'master' into feature/multi_cluster_search 2017-01-06 10:24:40 +01:00
Ryan Ernst eb596d7270 more renames 2017-01-06 01:03:45 -08:00
Ryan Ernst 6e406aed2d addressing more PR comments 2017-01-06 00:49:18 -08:00
Johannes Kanavin 27c57aeebe Fixed id's of 'worked example' in scripted metric aggs docs (#22430) 2017-01-05 14:37:27 -05:00
Jason Veatch 20f90178fe Docs: Detail on false/strict dynamic mapping setting (#22451)
Reference: https://www.elastic.co/guide/en/elasticsearch/guide/master/dynamic-mapping.html
2017-01-05 14:36:18 -05:00
javanna 5daf46286e remove ParseFieldMatcher usages from InternalSearchHit 2017-01-05 19:33:04 +01:00
javanna 975fee402a remove ParseFieldMatcher usages from suggesters 2017-01-05 19:33:04 +01:00
javanna 13dcb8ccbe remove ParseFieldMatcher usages from IngestMetadata 2017-01-05 19:33:04 +01:00
javanna d60e9bddd0 remove ParseFieldMatcher usages from IndexGraveyard 2017-01-05 19:33:04 +01:00
javanna d87a30647b remove ParseFieldMatcher usages from SearchAfterBuilder 2017-01-05 19:33:04 +01:00
javanna 723bdc4549 remove ParseFieldMatcher usages from FetchSourceContext 2017-01-05 19:33:04 +01:00
javanna dea7d65439 remove ParseFieldMatcher usages from RestSearchTemplateAction 2017-01-05 19:33:04 +01:00
javanna 6102523033 remove ParseFieldMatcher usages from Script parsing code 2017-01-05 19:33:04 +01:00
javanna 6b9a8db069 fix unchecked generics warnings in ObjectParser 2017-01-05 19:33:04 +01:00
javanna 1f7960aa52 ObjectParser to no longer require ParseFieldMatcherSupplier as its Context
ParseFieldMatcher as well as ParseFieldMatcherSupplier will be soon removed, hence the ObjectParser's context doesn't need to be a ParseFieldMatcherSupplier anymore. That will allow to remove ParseFieldMatcherSupplier's implementations, little by little.
2017-01-05 19:33:04 +01:00
javanna 9394792392 remove unused ParseFieldMatcher imports/arguments 2017-01-05 19:33:04 +01:00
Yannick Welsch 182e8115de [TEST] Fix IndexRecoveryIT.testDisconnectsDuringRecovery
The test currently checks that the recovering shard is not failed when it is not a primary relocation that has moved past the finalization step.
Checking if it has moved past that step is done by intercepting the request between the replication source and the target and checking if it has seen
then WAIT_FOR_CLUSTERSTATE action as this is the next action that is called after finalization. This action can, however, occur only after the shard was
already failed, and thus trip the assertion. This commit changes the check to look out for the FINALIZE action, independently of whether it succeeded or not.
2017-01-05 19:12:21 +01:00
Yannick Welsch cfc106d721 Don't close store under CancellableThreads (#22434)
#22325 changed the recovery retry logic to use unique recovery ids. The change also introduced an issue, however, which made it possible for the shard store to be closed under CancellableThreads, triggering assertions in the node locking logic. This commit limits the use of CancellableThreads only to the part where we wait on the old recovery target to be closed.
2017-01-05 18:11:58 +01:00
javanna ccc4e414ff remove double slash from license header 2017-01-05 17:15:34 +01:00
Simon Willnauer 349ea0f9b6 cut over to use : instead of | for cross cluster search 2017-01-05 17:03:12 +01:00
Simon Willnauer dca54734ac add basic docs 2017-01-05 16:10:34 +01:00
Simon Willnauer 0183b0c5a8 More cleanups 2017-01-05 15:23:55 +01:00
Simon Willnauer 1ef7115bbd Add javadocs to TransportActionProxy 2017-01-05 14:13:08 +01:00
Simon Willnauer 7b95c2f54c document and test concurrent remote cluster node discovery 2017-01-05 14:07:56 +01:00
Adrien Grand 97f3a9bd79 Relax LiveVersionMapTests.testRamBytesUsed.
With Java9's new restrictions we cannot compute ram usage as accurately as
before. See https://issues.apache.org/jira/browse/LUCENE-7595.
2017-01-05 11:33:24 +01:00
javanna 0a6827a5cc Make RestHighLevelClient non final 2017-01-05 11:29:41 +01:00
Nik Everett 232af512f4 Switch from standalone-test to standalone-rest-test
standalone-rest-test doesn't configure unit tests and for these
integ test only tests, that is what we want.
2017-01-05 10:55:47 +01:00
Nik Everett 812f63e5ef Require either BuildPlugin or StandaloneTestBasePlugin to use RestTestPlugin
It used to be that RestTestPlugin "came with" StandaloneTestBasePlugin
but we'd like to use it with BuildPlugin for the high level rest client.
2017-01-05 10:55:47 +01:00
javanna f0181b19f5 add REST high level client gradle submodule and first simple method
The RestHighLevelClient class takes as as an argument a low level client instance RestClient. The first method added is ping, which returns true if the call to HEAD / went ok and false if an IOException was thrown. Any other exception gets bubbled up.

There are two kinds of tests, a unit test (RestHighLevelClientTests) that verifies the interaction between high level and low level client, and an integration test (MainActionIT) which relies on an externally started es cluster to send requests to.
2017-01-05 10:55:47 +01:00
javanna 1899aea9ca [TEST] move randomHeaders method from RestClientTestCase to RestClientTestUtil and simplify headers assertions 2017-01-05 10:55:47 +01:00
Simon Willnauer 80bf01d3c0 Merge branch 'master' into feature/multi_cluster_search 2017-01-05 08:00:03 +01:00
Simon Willnauer a5daa5d3a2 Execute low level handshake in #openConnection (#22440)
Today we execute the low level handshake on the TCP layer in #connectToNode.
If #openConnection is used directly, which is truly expert, no handshake is executed
which allows connecting to nodes that are not necessarily compatible. This change
moves the handshake to #openConnection to prevent bypassing this logic.
2017-01-05 07:32:53 +01:00
Ryan Ernst fff26ba39e Use tests.asserts flag to allow disabling assertions
Randomized runner uses a flag, tests.asserts, which we have previously
not used, but is used in lucene for disabling assertions. This change
modifies the gradle configuration to look for this flag and pass through
to the test runner to determine whether -ea and -esa are added to the
java commandline for tests.
2017-01-04 15:56:53 -08:00
Ryan Ernst bf51522788 Add 5.3 version 2017-01-04 15:00:43 -08:00
Tim B be22a250b6 Replace Socket, ServerSocket, and HttpServer usages in tests with mocksocket versions (#22287)
This integrates the mocksocket jar with elasticsearch tests. Mocksocket wraps actions requiring SocketPermissions in doPrivilege blocks. This will eventually allow SocketPermissions to be assigned to the mocksocket jar opposed to the entire elasticsearch codebase.
2017-01-04 14:38:51 -06:00
Simon Willnauer daf1f53c39 Make RemoteClusterConnectionIT a unit test 2017-01-04 21:36:53 +01:00