Commit Graph

1546 Commits

Author SHA1 Message Date
Tanguy Leroux 9c03b4844f Merge branch 'master' into ccr
* master:
  992c7889ee Uncouple persistent task state and status (#31031)
  8c6ee7db54 Describe how to add a plugin in Dockerfile (#31340)
  1c5cec0ac7 Remove http status code maps (#31350)
  87a676e4d5 Do not set vm.max_map_count when unnecessary (#31285)
  e5b7137508 TEST: getCapturedRequestsAndClear should be atomic (#31312)
  0324103737 Painless: Fix bug for static method calls on interfaces (#31348)
  d6d0727aac QA: Fix resolution of default distribution (#31351)
  fcf1e41e42 Extract common http logic to server (#31311)
  6dd81ead74 Build: Fix the license in the pom zip and tar (#31336)
  8f886cd4be Treat ack timeout more like a publish timeout (#31303)
  9b293275af [ML] Add description to ML filters (#31330)
  f7a0cafe55 SQL: Fix build on Java 10
  375d09c588 [TEST] Fix RemoteClusterClientTests#testEnsureWeReconnect
  4877cec3e8 More detailed tracing when writing metadata (#31319)
  bbfe1eccc7 [Tests] Mutualize fixtures code in BaseHttpFixture (#31210)
2018-06-15 10:40:31 +02:00
Nhat Nguyen e5b7137508
TEST: getCapturedRequestsAndClear should be atomic (#31312)
We might lose messages between getCapturedRequestsAndClear calls.
This commit makes sure that both getCapturedRequestsAndClear and
getCapturedRequestsByTargetNodeAndClear are atomic.
2018-06-14 21:32:07 -04:00
Tim Brooks fcf1e41e42
Extract common http logic to server (#31311)
This is related to #28898. With the addition of the http nio transport,
we now have two different modules that provide http transports.
Currently most of the http logic lives at the module level. However,
some of this logic can live in server. In particular, some of the
setting of headers, cors, and pipelining. This commit begins this moving
in that direction by introducing lower level abstraction (HttpChannel,
HttpRequest, and HttpResonse) that is implemented by the modules. The
higher level rest request and rest channel work can live entirely in
server.
2018-06-14 15:10:02 -06:00
Nhat Nguyen 2cffd85f7f Merge branch 'master' into ccr
* master:
  Remove RestGetAllAliasesAction (#31308)
  Temporary fix for broken build
  Reenable Checkstyle's unused import rule (#31270)
  Remove remaining unused imports before merging #31270
  Fix non-REST doc snippet
  [DOC] Extend SQL docs
  Immediately flush channel after writing to buffer (#31301)
  [DOCS] Shortens ML API intros
  Use quotes in the call invocation (#31249)
  move security ingest processors to a sub ingest directory (#31306)
  Add 5.6.11 version constant.
  Fix version detection.
  SQL: Whitelist SQL utility class for better scripting (#30681)
  [Docs] All Rollup docs experimental, agg limitations, clarify DeleteJob (#31299)
  CCS: don't proxy requests for already connected node (#31273)
  Mute ScriptedMetricAggregatorTests testSelfReferencingAggStateAfterMap
  [test] opensuse packaging turn up debug logging
  Add unreleased version 6.3.1
  Removes experimental tag from scripted_metric aggregation (#31298)
  [Rollup] Metric config parser must use builder so validation runs (#31159)
  [ML] Check licence when datafeeds use cross cluster search  (#31247)
  Add notion of internal index settings (#31286)
  Test: Remove broken yml test feature (#31255)
  REST hl client: cluster health to default to cluster level (#31268)
  [ML] Update test thresholds to account for changes to memory control (#31289)
  Log warnings when cluster state publication failed to some nodes (#31233)
  Fix AntFixture waiting condition (#31272)
  Ignore numeric shard count if waiting for ALL (#31265)
  [ML] Implement new rules design (#31110)
  index_prefixes back-compat should test 6.3 (#30951)
  Core: Remove plain execute method on TransportAction (#30998)
  Update checkstyle to 8.10.1 (#31269)
  Set analyzer version in PreBuiltAnalyzerProviderFactory (#31202)
  Modify pipelining handlers to require full requests (#31280)
  Revert upgrade to Netty 4.1.25.Final (#31282)
  Use armored input stream for reading public key (#31229)
  Fix Netty 4 Server Transport tests. Again.
  REST hl client: adjust wait_for_active_shards param in cluster health (#31266)
  REST high-level Client: remove deprecated API methods (#31200)
  [DOCS] Mark SQL feature as experimental
  [DOCS] Updates machine learning custom URL screenshots (#31222)
  Fix naming conventions check for XPackTestCase
  Fix security Netty 4 transport tests
  Fix race in clear scroll (#31259)
  [DOCS] Clarify audit index settings when remote indexing (#30923)
  Delete typos in SAML docs (#31199)
  REST high-level client: add Cluster Health API (#29331)
  [ML][TEST] Mute tests using rules (#31204)
  Support RequestedAuthnContext (#31238)
  SyncedFlushResponse to implement ToXContentObject (#31155)
  Add Get Aliases API to the high-level REST client (#28799)
  Remove some line length supressions (#31209)
  Validate xContentType in PutWatchRequest. (#31088)
  [INGEST] Interrupt the current thread if evaluation grok expressions take too long (#31024)
  Suppress extras FS on caching directory tests
  Revert "[DOCS] Added 6.3 info & updated the upgrade table. (#30940)"
  Revert "Fix snippets in upgrade docs"
  Fix snippets in upgrade docs
  [DOCS] Added 6.3 info & updated the upgrade table. (#30940)
  LLClient: Support host selection (#30523)
  Upgrade to Netty 4.1.25.Final (#31232)
  Enable custom credentials for core REST tests (#31235)
  Move ESIndexLevelReplicationTestCase to test framework (#31243)
  Encapsulate Translog in Engine (#31220)
  HLRest: Add get index templates API (#31161)
  Remove all unused imports and fix CRLF (#31207)
  [Tests] Fix self-referencing tests
  [TEST] Fix testRecoveryAfterPrimaryPromotion
  [Docs] Remove mention pattern files in Grok processor (#31170)
  Use stronger write-once semantics for Azure repository (#30437)
  Don't swallow exceptions on replication (#31179)
  Limit the number of concurrent requests per node (#31206)
  Call ensureNoSelfReferences() on _agg state variable after scripted metric agg script executions (#31044)
  Move java version checker back to its own jar (#30708)
  [test] add fix for rare virtualbox error (#31212)
2018-06-14 10:47:36 -04:00
Tanguy Leroux bbfe1eccc7
[Tests] Mutualize fixtures code in BaseHttpFixture (#31210)
Many fixtures have similar code for writing the pid & ports files or
for handling HTTP requests. This commit adds an AbstractHttpFixture 
class in the test framework that can be extended for specific testing purposes.
2018-06-14 14:09:56 +02:00
Tanguy Leroux 4d7447cb5e
Reenable Checkstyle's unused import rule (#31270) 2018-06-14 09:52:46 +02:00
Nik Everett 77bb93557e
Test: Remove broken yml test feature (#31255)
The `requires_replica` yaml test feature hasn't worked for years. This
is what happens if you try to use it:
```
   > Throwable #1: java.lang.NullPointerException
   >    at __randomizedtesting.SeedInfo.seed([E6602FB306244B12:6E341069A8D826EA]:0)
   >    at org.elasticsearch.test.rest.yaml.Features.areAllSupported(Features.java:58)
   >    at org.elasticsearch.test.rest.yaml.section.SkipSection.skip(SkipSection.java:144)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:321)
```

None of our tests use it.
2018-06-13 09:33:06 -04:00
Tanguy Leroux 8b4d80ad09
Fix AntFixture waiting condition (#31272)
The AntFixture waiting condition is evaluated to false 
but it should be true.
2018-06-13 12:40:22 +02:00
Ryan Ernst a65b18f19d Core: Remove plain execute method on TransportAction (#30998)
TransportAction has many variants of execute. One of those variants
executes by returning a future, which is then often blocked on by
calling get(). This commit removes this variant of execute, instead
using a helper method for tests that want to block, or having tests
pass in a PlainActionFuture directly as a listener.

Co-authored-by: Simon Willnauer <simonw@apache.org>
2018-06-13 09:58:13 +02:00
Van0SS d5e8a5cd69 REST high-level client: add Cluster Health API (#29331)
Relates to #27205
2018-06-12 13:34:06 +02:00
olcbean 7d7ead95b2 Add Get Aliases API to the high-level REST client (#28799)
Given the weirdness of the response returned by the get alias API, we went for a client specific response, which allows us to hold the error message, exception and status returned as part of the response together with aliases. See #30536 .

Relates to #27205
2018-06-12 10:26:17 +02:00
Nik Everett 0d9b78834f
LLClient: Support host selection (#30523)
Allows users of the Low Level REST client to specify which hosts a
request should be run on. They implement the  `NodeSelector` interface
or reuse a built in selector like `NOT_MASTER_ONLY` to chose which nodes
are valid. Using it looks like:
```
Request request = new Request("POST", "/foo/_search");
RequestOptions options = request.getOptions().toBuilder();
options.setNodeSelector(NodeSelector.NOT_MASTER_ONLY);
request.setOptions(options);
...
```

This introduces a new `Node` object which contains a `HttpHost` and the
metadata about the host. At this point that metadata is just `version`
and `roles` but I plan to add node attributes in a followup. The
canonical way to **get** this metadata is to use the `Sniffer` to pull
the information from the Elasticsearch cluster.

I've marked this as "breaking-java" because it breaks custom
implementations of `HostsSniffer` by renaming the interface to
`NodesSniffer` and by changing it from returning a `List<HttpHost>` to a
`List<Node>`. It *shouldn't* break anyone else though.

Because we expect to find it useful, this also implements `host_selector`
support to `do` statements in the yaml tests. Using it looks a little
like:

```
---
"example test":
  - skip:
      features: host_selector
  - do:
      host_selector:
        version: " - 7.0.0" # same syntax as skip
      apiname:
        something: true
```

The `do` section parses the `version` string into a host selector that
uses the same version comparison logic as the `skip` section. When the
`do` section is executed it passed the off to the `RestClient`, using
the `ElasticsearchHostsSniffer` to sniff the required metadata.

The idea is to use this in mixed version tests to target a specific
version of Elasticsearch so we can be sure about the deprecation
logging though we don't currently have any examples that need it. We do,
however, have at least one open pull request that requires something
like this to properly test it.

Closes #21888
2018-06-11 17:07:27 -04:00
Nhat Nguyen dda56fc0fc
Move ESIndexLevelReplicationTestCase to test framework (#31243)
Other components might benefit from the testing infra provided by
ESIndexLevelReplicationTestCase. This commit moves it to the test
framework.
2018-06-11 12:47:38 -04:00
Lee Hinman c064b507df
Encapsulate Translog in Engine (#31220)
This removes the abstract `getTranslog` method in `Engine`, instead leaving it
to the abstract implementations of the other methods that use the translog. This
allows future Engines not to have a Translog, as instead they must implement the
methods that use the translog pieces to return necessary values.
2018-06-11 09:44:50 -06:00
Tanguy Leroux bf58660482
Remove all unused imports and fix CRLF (#31207)
The X-Pack opening and the recent other refactorings left a lot of 
unused imports in the codebase. This commit removes them all.
2018-06-11 15:12:12 +02:00
Nhat Nguyen 9feff9809b CCR side of #31213
Relates #31213
2018-06-09 20:04:45 -04:00
Nhat Nguyen afe02a3cac Merge branch 'master' into ccr
* master:
  Move default location of dependencies report (#31228)
  Remove dependencies report task dependencies (#31227)
  Add recognition of MPL 2.0 (#31226)
  Fix unknown licenses (#31223)
  Remove version from license file name for GCS SDK (#31221)
  Fully encapsulate LocalCheckpointTracker inside of the engine (#31213)
  [DOCS] Added 'fail_on_unsupported_field' param to MLT. Closes #28008 (#31160)
  Add licenses for transport-nio (#31218)
  Remove DocumentFieldMappers#simpleMatchToFullName. (#31041)
  Allow to trim all ops above a certain seq# with a term lower than X, post backport fix (#31211)
  Compliant SAML Response destination check (#31175)
  Remove DocumentFieldMappers#smartNameFieldMapper, as it is no longer needed. (#31018)
  Remove extraneous references to 'tokenized' in the mapper code. (#31010)
  Allow to trim all ops above a certain seq# with a term lower than X (#30176)
  SQL: Make a single JDBC driver jar (#31012)
  Enhance license detection for various licenses (#31198)
  [DOCS] Add note about long-lived idle connections (#30990)
  Move number of language analyzers to analysis-common module (#31143)
  Default max concurrent search req. numNodes * 5 (#31171)
  flush job to ensure all results have been written (#31187)
2018-06-09 19:53:21 -04:00
Jason Tedor 65c107b47d
Fix unknown licenses (#31223)
The goal of this commit is to address unknown licenses when producing
the dependencies info report. We have two different checks that we run
on licenses. The first check is whether or not we have stashed a copy of
the license text for a dependency in the repository. The second is to
map every dependency to a license type (e.g., BSD 3-clause). The problem
here is that the way we were handling licenses in the second check
differs from how we handle licenses in the first check. The first check
works by finding a license file with the name of the artifact followed
by the text -LICENSE.txt. Yet in some cases we allow mapping an artifact
name to another name used to check for the license (e.g., we map
lucene-.* to lucene, and opensaml-.* to shibboleth. The second check
understood the first way of looking for a license file but not the
second way. So in this commit we teach the second check about the
mappings from artifact names to license names. We do this by copying the
configuration from the dependencyLicenses task to the dependenciesInfo
task and then reusing the code from the first check in the second
check. There were some other challenges here though. For example,
dependenciesInfo was checking too many dependencies. For now, we should
only be checking direct dependencies and leaving transitive dependencies
from another org.elasticsearch artifact to that artifact (we want to do
this differently in a follow-up). We also want to disable
dependenciesInfo for projects that we do not publish, users only care
about licenses they might be exposed to if they use our assembled
products. With all of the changes in this commit we have eliminated all
unknown licenses. A follow-up will enforce that when we add a new
dependency it does not get mapped to unknown, these will be forbidden in
the future. Therefore, with this change and earlier changes are left
having no unknown licenses and two custom licenses; custom here means it
does not map to an SPDX license type. Those two licenses are xz and
ldapsdk. A future change will not allow additional custom licenses
unless they are explicitly whitelisted. This ensures that if a new
dependency is added it is mapped to an SPDX license or mapped to custom
because it does not have an SPDX license.
2018-06-09 07:28:41 -04:00
Lee Hinman bdb0fb2555
Fully encapsulate LocalCheckpointTracker inside of the engine (#31213)
* Fully encapsulate LocalCheckpointTracker inside of the engine

This makes the Engine interface not expose the `LocalCheckpointTracker`, instead
exposing the pieces needed (like retrieving the local checkpoint) as individual
methods.
2018-06-08 17:19:41 -06:00
Julie Tibshirani 00b0e10063
Remove DocumentFieldMappers#simpleMatchToFullName. (#31041)
* Remove DocumentFieldMappers#simpleMatchToFullName, as it is duplicative of MapperService#simpleMatchToIndexNames.
* Rename MapperService#simpleMatchToIndexNames -> simpleMatchToFullName for consistency.
* Simplify EsIntegTestCase#assertConcreteMappingsOnAll to accept concrete fields instead of wildcard patterns.
2018-06-08 13:53:35 -07:00
Jason Tedor 64b4cdeda6
Merge remote-tracking branch 'elastic/master' into ccr
* elastic/master: (53 commits)
  Painless: Restructure/Clean Up of Spec Documentation (#31013)
  Update ignore_unmapped serialization after backport
  Add back dropped substitution on merge
  high level REST api: cancel task (#30745)
  Enable engine factory to be pluggable (#31183)
  Remove vestiges of animal sniffer (#31178)
  Rename elasticsearch-nio to nio (#31186)
  Rename elasticsearch-core to core (#31185)
  Move cli sub-project out of server to libs (#31184)
  [DOCS] Fixes broken link in auditing settings
  QA: Better seed nodes for rolling restart
  [DOCS] Moves ML content to stack-docs
  [DOCS] Clarifies recommendation for audit index output type (#31146)
  Add nio-transport as option for http smoke tests (#31162)
  QA: Set better node names on rolling restart tests
  Add support for ignore_unmapped to geo sort (#31153)
  Share common parser in some AcknowledgedResponses (#31169)
  Fix random failure on SearchQueryIT#testTermExpansionExceptionOnSpanFailure
  Remove reference to multiple fields with one name (#31127)
  Remove BlobContainer.move() method (#31100)
  ...
2018-06-07 23:33:42 -04:00
Jason Tedor e481b860a1
Enable engine factory to be pluggable (#31183)
This commit enables the engine factory to be pluggable based on index
settings used when creating the index service for an index.
2018-06-07 17:01:06 -04:00
Tanguy Leroux b5f05f676c
Remove BlobContainer.move() method (#31100)
closes #30680
2018-06-07 10:48:31 +02:00
Simon Willnauer 5c6711b8a4
Use a `_recovery_source` if source is omitted or modified (#31106)
Today if a user omits the `_source` entirely or modifies the source
on indexing we have no chance to re-create the document after it has
been added. This is an issue for CCR and recovery based on soft deletes
which we are going to make the default. This change adds an additional
recovery source if the source is disabled or modified that is only kept
around until the document leaves the retention policy window.

This change adds a merge policy that efficiently removes this extra source
on merge for all document that are live and not in the retention policy window
anymore.
2018-06-07 07:39:28 +02:00
Tim Brooks 67e73b4df4
Combine accepting selector and socket selector (#31115)
This is related to #27260. This commit combines the AcceptingSelector
and SocketSelector classes into a single NioSelector. This change
allows the same selector to handle both server and socket channels. This
is valuable as we do not necessarily want a dedicated thread running for
accepting channels.

With this change, this commit removes the configuration for dedicated
accepting selectors for the normal transport class. The accepting
workload for new node connections is likely low, meaning that there is
no need to dedicate a thread to this process.
2018-06-06 11:59:54 -06:00
Jason Tedor bf1152fcc6
Use follower primary term when applying operations (#31113)
The primary shard copy on the following has authority of the replication
operations that occur on the following side in cross-cluster
replication. Yet today we are using the primary term directly from the
operations on the leader side. Instead we should be replacing the
primary term on the following side with the primary term of the primary
on the following side. This commit does this by copying the translog
operations with the corrected primary term. This ensures that we use
this primary term while applying the operations on the primary, and when
replicating them across to the replica (where the replica request was
carrying the primary term of the primary shard copy on the follower).
2018-06-06 11:03:57 -04:00
Yannick Welsch 1dca00deb9
Remove extra checks from HdfsBlobContainer (#31126)
This commit saves one network roundtrip when reading or deleting files from an HDFS repository.
2018-06-06 16:38:37 +02:00
Nhat Nguyen 755a25a41e
Replace exact numDocs by soft-del count in SegmentInfo (#31086)
This PR adapts/utilizes recent enhancements in Lucene-7.4:

- Replaces exactNumDocs by the soft-deletes count in SegmentCommitInfo.
This enhancement allows us to back out changes introduced in #30228.

- Always configure the soft-deletes field in IWC
2018-06-05 14:11:19 -04:00
Nhat Nguyen 2b5c0d77e9 Merge branch 'master' into ccr
* master:
  Removing erroneous repeat
  Adapt bwc versions after backporting #30983 to 6.4
  [Tests] Muting RatedRequestsTests#testXContentParsingIsNotLenient
  TEST:  Retry synced-flush if ongoing ops on primary (#30978)
  Fix docs build.
  Only auto-update license signature if all nodes ready (#30859)
  Add BlobContainer.writeBlobAtomic() (#30902)
  Add a doc value format to binary fields. (#30860)
2018-06-05 10:58:35 -04:00
Tanguy Leroux 9531b7bbcb
Add BlobContainer.writeBlobAtomic() (#30902)
This commit adds a new writeBlobAtomic() method to the BlobContainer
interface that can be implemented by repository implementations which
support atomic writes operations.

When the BlobContainer implementation does not provide a specific 
implementation of writeBlobAtomic(), then the writeBlob() method is used.

Related to #30680
2018-06-05 13:00:43 +02:00
Nhat Nguyen 646e1c01b0 Merge branch 'master' into ccr
* master:
  Add get mappings support to high-level rest client (#30889)
  Fix index prefixes to work with span_multi (#31066)
  [DOCS] Removes redundant authorization pages
  [DOCS] Re-adds custom realm
  Change ObjectParser exception (#31030)
  Upgrade to Lucene-7.4.0-snapshot-0a7c3f462f (#31073)
2018-06-04 16:32:43 -04:00
Nhat Nguyen d9a0a02e89 Merge branch 'master' into ccr
* master:
  Match phrase queries against non-indexed fields should throw an exception (#31060)
  In the internal highlighter APIs, use the field type as opposed to the mapper. (#31039)
  [DOCS] Removes duplicated authentication pages
  Enable customizing REST tests blacklist (#31074)
  Make sure KeywordFieldMapper#clone preserves split_queries_on_whitespace. (#31049)
  [DOCS] Moves machine learning overview to stack-docs
  [ML] Add secondary sort to ML events (#31063)
  [Rollup] Specialize validation exception for easier management (#30339)
  Adapt bwc versions after backporting #31045 to 6.3
  Remove usage of explicit type in docs (#29667)
  Share common readFrom/writeTo code in AcknowledgeResponse (#30983)
  Adapt bwc versions after backporting #31045 to 6.x
  Mute MatchPhrase*QueryBuilderTests
  [Docs] Fix typo in watcher conditions documentation (#30989)
  Remove wrong link in index phrases doc
  Move pipeline APIs to ingest namespace (#31027)
  [DOCS] Fixes accounting setting names (#30863)
  [DOCS] Rewords _field_names documentation (#31029)
  Index phrases (#30450)
  Remove leftover debugging from PTCMDT
  Fix PTCMDT#testMinVersionSerialization
  Make Persistent Tasks implementations version and feature aware (#31045)
2018-06-04 14:32:28 -04:00
Christoph Büscher 3f87c79500
Change ObjectParser exception (#31030)
ObjectParser should throw XContentParseExceptions, not IAE. A dedicated parsing
exception can includes the place where the error occurred.

Closes #30605
2018-06-04 20:20:37 +02:00
Jason Tedor be55da18c2
Enable customizing REST tests blacklist (#31074)
This commit enables adding additional REST tests to the blacklist for
builds that already define tests.rest.blacklist.
2018-06-04 13:35:49 -04:00
Boaz Leskes a7ceefe93f
Make Persistent Tasks implementations version and feature aware (#31045)
With #31020 we introduced the ability for transport clients to indicate what features they support
in order to make sure we don't serialize object to them they don't support. This PR adapts the
serialization logic of persistent tasks to be aware of those features and not serialize tasks that
aren't supported. 

Also, a version check is added for the future where we may add new tasks implementations and
need to be able to indicate they shouldn't be serialized both to nodes and clients.

As the implementation relies on the interface of `PersistentTaskParams`, these are no longer
optional. That's acceptable as all current implementation have them and we plan to make
`PersistentTaskParams` more central in the future.

Relates to #30731
2018-06-03 21:51:08 +02:00
Nhat Nguyen 5fd759b693 Merge branch 'master' into ccr
* master:
  Adapt transport tests for the extra byte introduced in #31020
2018-06-02 08:23:19 -04:00
Boaz Leskes 65d3f0efca Adapt transport tests for the extra byte introduced in #31020
We now serialize a feature array, which takes an extra byte when empty.
2018-06-02 13:04:40 +02:00
Nhat Nguyen a527095b8d Merge branch 'master' into ccr
* master:
  Avoid randomization bug in FeatureAwareTests
  Adjust BWC version on client features
  Add TRACE, CONNECT, and PATCH http methods (#31035)
  Adjust BWC version on client features
  [DOCS] Make geoshape docs less memory hungry (#31014)
  Fix handling of percent-encoded spaces in Windows batch files (#31034)
  [Docs] Fix a typo in Create Index naming limitation (#30891)
  Introduce client feature tracking (#31020)
  Ensure that index_prefixes settings cannot be changed (#30967)
  REST high-level client: add delete ingest pipeline API (#30865)
  [ML][TEST] Fix bucket count assertion in all tests in ModelPlotsIT (#31026)
  Allow rollup job creation only if cluster is x-pack ready (#30963)
  Fix interoperability with < 6.3 transport clients (#30971)
  Add an option to split keyword field on whitespace at query time (#30691)
  [Tests] Fix alias names in PutIndexTemplateRequestTests (#30960)
  REST high-level client: add get ingest pipeline API (#30847)
  Cross Cluster Search: preserve remote status code (#30976)
  High-level client: list tasks failure to not lose nodeId (#31001)
  [DOCS] Fixes links (#31011)
  Watcher: Give test a little more time
  Reuse expiration date of trial licenses (#30950)
  Remove unused query methods from MappedFieldType. (#30987)
  Transport client: Don't validate node in handshake (#30737)
  [DOCS] Clarify not all PKCS12 usable as truststores (#30750)
  HLRest: Allow caller to set per request options (#30490)
  Remove version read/write logic in Verify Response (#30879)
  [DOCS] Update readme for testing x-pack code snippets (#30696)
  Ensure intended key is selected in SamlAuthenticatorTests (#30993)
  Core: Remove RequestBuilder from Action (#30966)
2018-06-01 23:09:29 -04:00
Jason Tedor 4522b57e07
Introduce client feature tracking (#31020)
This commit introduces the ability for a client to communicate to the
server features that it can support and for these features to be used in
influencing the decisions that the server makes when communicating with
the client. To this end we carry the features from the client to the
underlying stream as we carry the version of the client today. This
enables us to enhance the logic where we make protocol decisions on the
basis of the version on the stream to also make protocol decisions on
the basis of the features on the stream. With such functionality, the
client can communicate to the server if it is a transport client, or if
it has, for example, X-Pack installed. This enables us to support
rolling upgrades from the OSS distribution to the default distribution
without breaking client connectivity as we can now elect to serialize
customs in the cluster state depending on whether or not the client
reports to us using the feature capabilities that it can under these
customs. This means that we would avoid sending a client pieces of the
cluster state that it can not understand. However, we want to take care
and always send the full cluster state during node-to-node communication
as otherwise we would end up with different understanding of what is in
the cluster state across nodes depending on which features they reported
to have. This is why when deciding whether or not to write out a custom
we always send the custom if the client is not a transport client and
otherwise do not send the custom if the client is transport client that
does not report to have the feature required by the custom.

Co-authored-by: Yannick Welsch <yannick@welsch.lu>
2018-06-01 11:45:35 -04:00
Jim Ferenczi 0791f93dbd
Add an option to split keyword field on whitespace at query time (#30691)
This change adds an option named `split_queries_on_whitespace` to the `keyword`
field type. When set to true full text queries (`match`, `multi_match`, `query_string`, ...) that target the field will split the input on whitespace to build the query terms. Defaults to `false`.
Closes #30393
2018-06-01 09:47:03 +02:00
Nik Everett b225f5e5c6
HLRest: Allow caller to set per request options (#30490)
This modifies the high level rest client to allow calling code to
customize per request options for the bulk API. You do the actual
customization by passing a `RequestOptions` object to the API call
which is set on the `Request` that is generated by the high level
client. It also makes the `RequestOptions` a thing in the low level
rest client. For now that just means you use it to customize the
headers and the `httpAsyncResponseConsumerFactory` and we'll add
node selectors and per request timeouts in a follow up.

I only implemented this on the bulk API because it is the first one
in the list alphabetically and I wanted to keep the change small
enough to review. I'll convert the remaining APIs in a followup.
2018-05-31 13:59:52 -04:00
Ryan Ernst 46e8d97813
Core: Remove RequestBuilder from Action (#30966)
This commit removes the RequestBuilder generic type from Action. It was
needed to be used by the newRequest method, which in turn was used by
client.prepareExecute. Both of these methods are now removed, along with
the existing users of prepareExecute constructing the appropriate
builder directly.
2018-05-31 16:15:00 +02:00
Martijn van Groningen 56472d6505
Merge remote-tracking branch 'es/master' into ccr
* es/master: (24 commits)
  Add missing_bucket option in the composite agg (#29465)
  Rename index_prefix to index_prefixes (#30932)
  Rename methods in PersistentTasksService (#30837)
  [DOCS] Fix watcher file location
  Update the version checks around range bucket keys, now that the change was backported.
  Use dedicated ML APIs in tests (#30941)
  [DOCS] Remove reference to platinum Docker image (#30916)
  Minor clean-up in InternalRange. (#30886)
  stable filemode for zip distributions (#30854)
  [DOCS] Adds missing TLS settings for auditing (#30822)
  [test] packaging: use shell when running commands (#30852)
  Fix location of AbstractHttpServerTransport (#30888)
  [test] packaging test logging for suse distros
  Moved keyword tokenizer to analysis-common module (#30642)
  Upgrade to Lucene-7.4-snapshot-1cbadda4d3 (#30928)
  Limit the scope of BouncyCastle dependency (#30358)
  [DOCS] Reset edit links (#30909)
  Fix IndexTemplateMetaData parsing from xContent (#30917)
  Remove log traces in AzureStorageServiceImpl and fix test (#30924)
  Deprecate accepting malformed requests in stored script API (#28939)
  ...
2018-05-30 09:55:34 +02:00
Martijn van Groningen 544822c78b
Moved keyword tokenizer to analysis-common module (#30642)
Relates to #23658
2018-05-29 19:22:28 +02:00
Nhat Nguyen 03e3bd28c9 Merge branch 'master' into ccr
* master:
  silence InstallPluginCommandTests, see https://github.com/elastic/elasticsearch/issues/30900
  Remove left-over comment
  Fix double semicolon in import statement
  [TEST] Fix minor random bug from #30794
  Include size of snapshot in snapshot metadata #18543, bwc clean up (#30890)
  Enabling testing against an external cluster (#30885)
  Add public key header/footer (#30877)
  SQL: Remove the last remaining server dependencies from jdbc (#30771)
  Include size of snapshot in snapshot metadata (#29602)
  Do not serialize basic license exp in x-pack info (#30848)
  Change BWC version for VerifyRepositoryResponse (#30796)
  [DOCS] Document index name limitations (#30826)
  Harmonize include_defaults tests (#30700)
2018-05-28 12:16:20 -04:00
Vladimir Dolzhenko 81eb8ba0f0
Include size of snapshot in snapshot metadata (#29602)
Include size of snapshot in snapshot metadata

Adds difference of number of files (and file sizes) between prev and current snapshot. Total number/size reflects total number/size of files in snapshot.

Closes #18543
2018-05-25 21:04:50 +02:00
Martijn van Groningen e6dd4144ce
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  Move score script context from SearchScript to its own class (#30816)
  Fix bad version check writing Repository nodes (#30846)
  [docs] explainer for java packaging tests (#30825)
  Remove Throwable usage from transport modules (#30845)
  REST high-level client: add put ingest pipeline API (#30793)
  Update the version checks around ip_range bucket keys, now that the change was backported.
  Mute IndexMasterFailoverIT.testMasterFailoverDuringIndexingWithMappingChanges
  Use geohash cell instead of just a corner in geo_bounding_box (#30698)
  Limit user to single concurrent auth per realm (#30794)
  [Tests] Move templated _rank_eval tests (#30679)
  Security: fix dynamic mapping updates with aliases (#30787)
  Ensure that ip_range aggregations always return bucket keys. (#30701)
  Use remote client in TransportFieldCapsAction (#30838)
  Move Watcher versioning setting to meta field (#30832)
  [Docs] Explain incomplete dates in range queries (#30689)
  Move persistent task registrations to core (#30755)
  Decouple ClusterStateTaskListener & ClusterApplier (#30809)
  Send client headers from TransportClient (#30803)
  Packaging: Ensure upgrade_is_oss flag file is always deleted (#30732)
  Force stable file modes for built packages (#30823)
2018-05-25 11:07:45 +02:00
Martijn van Groningen ae2f021f1c
Move score script context from SearchScript to its own class (#30816) 2018-05-25 07:17:50 +02:00
Tim Brooks e8b70273c1
Remove Throwable usage from transport modules (#30845)
Currently nio and netty modules use the CompletableFuture class for
managing listeners. This is unfortunate as that class accepts
Throwable. This commit adds a class CompletableContext that wraps
the CompletableFuture but does not accept Throwable. This allows the
modification of netty and nio logic to no longer handle Throwable.
2018-05-24 17:33:29 -06:00
David Turner ff0b6c795a
Decouple ClusterStateTaskListener & ClusterApplier (#30809)
Today, the `ClusterApplier` and `MasterService` both use the
`ClusterStateTaskListener` interface to notify their callers when asynchronous
activities have completed. However, this is not wholly appropriate: none of the
callers into the `ClusterApplier` care about the `ClusterState` arguments that
they receive.  This change introduces a dedicated ClusterApplyListener
interface for callers into the `ClusterApplier`, to distinguish these listeners
from the real `ClusterStateTaskListener`s that are waiting for responses from
the `MasterService`.
2018-05-24 09:05:09 +01:00