Commit Graph

40499 Commits

Author SHA1 Message Date
Jack Conradson a381749aac
Painless: Fix Semicolon Regression (#33212)
Trailers (statements following something like an if statement) that don't use brackets currently require a semicolon even if they're the last statement. This is a regression caused by (#29566) and noted by (#33193). This change fixes the regression and adds a test for the broken case.
2018-08-28 12:40:41 -07:00
Jake Landis e9b0807c67
ingest: minor - update test to include dissect (#33211)
This change also includes placing the bytes processor in the correct
order (helps to avoid merge conflict when back patching processors)
2018-08-28 11:55:04 -07:00
Nik Everett 6c8f568808
Switch remaining LLREST usage to new style Requests (#33171)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. In a
long series of PRs I've changed all of the old style requests that I
could find with `grep`. In this PR I change all requests that I could
find by *removing* the deprecated methods. Since this is a non-trivial
change I do not include actually removing the deprecated requests. I'll
do that in a follow up. But this should be the last set of usage
removals before the actual deprecated method removal. Yay!
2018-08-28 14:20:14 -04:00
Sohaib Iftikhar 7f5e29ddb2 HLREST: add reindex API (#32679)
Adds the reindex API to the high level REST client.
2018-08-28 13:02:23 -04:00
Zachary Tong 353112a033
[Rollup] Better error message when trying to set non-rollup index (#32965)
We don't allow the user to configure a rollup index against an
existing index, but the exceptions that we return are not clear about
that.  They indicate issues with metadata, instead of stating
the real reason (not allowed to use a non-rollup index to store
rollup data).

This makes the exception better, and adds a bit more testing
2018-08-28 11:50:35 -04:00
Michael Basnight 5f0f990afd
HLRC: Use Optional in validation logic (#33104)
The Validatable class comes from an old class in server code, that
assumed null was returned in the event of validation having no
errors. This commit changes that to use Optional, which is cleaner than
passing around null objects.
2018-08-28 10:33:18 -05:00
Jay Modi 1e11b05b58
Remove unused User class from protocol (#33137)
This commit removes the unused User class from the protocol project.
This class was originally moved into protocol in preparation for moving
more request and response classes, but given the change in direction
for the HLRC this is no longer needed. Additionally, this change also
changes the package name for the User object in x-pack/plugin/core to
its original name.
2018-08-28 08:55:29 -06:00
Jake Landis 79b507dbf5
ingest: Introduce the dissect processor (#32884)
* ingest: Introduce the dissect processor

The ingest node dissect processor is an alternative to Grok
to split a string based on a pattern. Dissect differs from
Grok such that regular expressions are not used to split the
string.

Dissect can be used to parse a source text field with a
simpler pattern, and is often faster the Grok for basic string
parsing. This processor uses the dissect library which
does most of the work.
2018-08-28 07:11:20 -07:00
Jilles van Gurp d7e4a49355 [Docs] Add link to es-kotlin-wrapper-client (#32618)
ES Kotlin Wrapper client is a library that wraps the official Highlevel Elasticsearch HTTP client for Java.
2018-08-28 14:25:55 +02:00
lipsill b7c0d2830a [Docs] Remove repeating words (#33087) 2018-08-28 13:16:43 +02:00
Dominik 525cda0331 Minor spelling and grammar fix (#32931) 2018-08-28 12:50:19 +02:00
Jonathan Little 9d92a87ae6 Remove support for deprecated params._agg/_aggs for scripted metric aggregations (#32979) 2018-08-28 09:27:43 +01:00
Alexander Reelsen 19ef41ee82
Watcher: Simplify finding next date in cron schedule (#33015)
The code introduced in 3fa36807f8 to fix
an issue with crons always returning -1 was not very readable. This
implementation uses streams to improve readability.
2018-08-28 09:06:43 +02:00
Alpar Torok 2cc611604f
Run Third party audit with forbidden APIs CLI (part3/3) (#33052)
The new implementation is functional equivalent with the old, ant based one.
It parses task standard error to get the missing classes and violations in the same way.
I considered re-using ForbiddenApisCliTask but Gradle makes it hard to build inheritance with tasks that have task actions , since the order of the task actions can't be controlled.
This inheritance isn't dully desired either as the third party audit task is much more opinionated and we don't want to expose some of the configuration.
We could probably extract a common base class without any task actions, but probably more trouble than it's worth.

Closes #31715
2018-08-28 10:03:30 +03:00
Alpar Torok 71d5c66fd3
Fix plugin build test on Windows (#33078)
Fix plugin build test on Windows
2018-08-28 10:00:11 +03:00
Armin Braun daee8bd133
HLRC+MINOR: Remove Unused Private Method (#33165)
* This one seems to be unused since 92eb32477
2018-08-28 07:44:01 +02:00
Jonathan Little 29ba143e2c Remove old unused test script files (#32970) 2018-08-27 16:29:50 -07:00
Jason Tedor 5937e499e1
Build analysis-icu client JAR (#33184)
This plugin needs to be able to be installed client side because it
contains doc values formats that can be returned to the transport
client. To keep this simple for developers, we publish the client JAR to
Maven so that they can depend on the plugin in their POM and install the
plugin there.
2018-08-27 16:41:32 -04:00
Nhat Nguyen 014b3236dc
Ensure to generate identical NoOp for the same failure (#33141)
We generate slightly different NoOps in InternalEngine and
TransportShardBulkAction for the same failure.

1. InternalEngine uses Exception#getFailure to generate a message
without the class name: newOp [NoOp{seqNo=1, primaryTerm=1,
reason='Contexts are mandatory in context enabled completion field
[suggest_context]'}].

2. TransportShardBulkAction uses Exception#toString to generate a
message with the class name: NoOp{seqNo=1, primaryTerm=1,
reason='java.lang.IllegalArgumentException: Contexts are mandatory in
context enabled completion field [suggest_context]'}.

If a write operation fails while a replica is recovering, that replica
will possibly receive two different NoOps: one from recovery and one
from replication. These two different NoOps will trip
TranslogWriter#assertNoSeqNumberConflict assertion.

This commit ensures that we generate the same Noop for the same failure.

Closes #32986
2018-08-27 15:59:42 -04:00
Luca Cavanna ed0571e16c
ShardSearchFailure#readFrom to set index and shardId (#33161)
As part of recent changes made to `ShardOperationFailedException` we introduced `index` and `shardId` members to the base class, but the subclasses are entirely responsible for the serialization of such fields. In the case of `ShardSearchFailure`, we have an additional `SearchShardTarget` instance member which also holds the index and the shardId, hence they get serialized as part of `SearchShardTarget` itself. When de-serializing a `ShardSearchFailure` though, we need to remember to also set the parent class `index` and `shardId` fields otherwise they get lost

Relates to #32640
2018-08-27 20:31:27 +02:00
Jason Tedor 318df2a107
Adjust BWC version on mapping version
The introduction of mapping version on index metadata has been
backported to 6.x. This commit adjusts the BWC version around mapping
version to account for this backport.
2018-08-27 13:17:15 -04:00
Jay Modi 5d9c270608
Token API supports the client_credentials grant (#33106)
This change adds support for the client credentials grant type to the
token api. The client credentials grant allows for a client to
authenticate with the authorization server and obtain a token to access
as itself. Per RFC 6749, a refresh token should not be included with
the access token and as such a refresh token is not issued when the
client credentials grant is used.

The addition of the client credentials grant will allow users
authenticated with mechanisms such as kerberos or PKI to obtain a token
that can be used for subsequent access.
2018-08-27 10:56:21 -06:00
Jay Modi 309fb22181
Build: forked compiler max memory matches jvmArgs (#33138)
This commit removes the setting of the fork options maximum memory size
in our build plugin and instead adds the value in the gradle.properties
file to be alongside the value set in jvmArgs.

This change is necessary when using parallel compilation as 512m is not
sufficient for parallel compilation on some machines.
2018-08-27 10:26:25 -06:00
Jason Tedor 2aef7e0900
Introduce mapping version to index metadata (#33147)
This commit introduces mapping version to index metadata. This value is
monotonically increasing and is updated on mapping updates. This will be
useful in cross-cluster replication so that we can request mapping
updates from the leader only when there is a mapping update as opposed
to the strategy we employ today which is to request a mapping update any
time there is an index metadata update. As index metadata updates can
occur for many reasons other than mapping updates, this leads to some
unnecessary requests and work in cross-cluster replication.
2018-08-27 12:21:11 -04:00
Andrei Stefan 3d9ca4baee
SQL: Enable aggregations to create a separate bucket for missing values (#32832)
Enable aggregations to create a separate bucket for missing values.
2018-08-27 17:56:28 +03:00
Jason Tedor c41c614527
Fix grammar in contributing docs
This commit fixes an instance of odd comma placement in the contributing
docs.
2018-08-27 10:17:42 -04:00
Armin Braun f7a9186372
SECURITY: Fix Compile Error in ReservedRealmTests (#33166)
* This was broken by #32515 since the 5.x versions
were removed between PR creation and merge
2018-08-27 15:08:27 +02:00
Shaunak Kashyap 1779d3376a
APM server monitoring (#32515)
* Adding new MonitoredSystem for APM server

* Teaching Monitoring template utils about APM server monitoring indices

* Documenting new monitoring index for APM server

* Adding monitoring index template for APM server

* Copy pasta typo

* Removing metrics.libbeat.config section from mapping

* Adding built-in user and role for APM server user

* Actually define the role :)

* Adding missing import

* Removing index template and system ID for apm server

* Shortening line lengths

* Updating expected number of built-in users in integration test

* Removing "system" from role and user names

* Rearranging users to make tests pass
2018-08-27 08:42:40 -04:00
Mikita Karaliou f1f6d4ed33 Support only string `format` in date, root object & date range (#28117)
Limit date `format` attribute to String values only.

Closes #23650
2018-08-27 12:24:51 +02:00
Tanguy Leroux e1e8cf382f
[Rollup] Move toBuilders() methods out of rollup config objects (#32585) 2018-08-27 09:18:26 +02:00
Alpar Torok 974f839093
Fix forbiddenapis on java 11 (#33116)
Cap forbiddenapis to java version 10
2018-08-27 08:47:42 +03:00
Alpar Torok 30c3b36395
Apply publishing to genreate pom (#33094) 2018-08-27 08:44:06 +03:00
Daniel Mitterdorfer 06c0055c0f
Have circuit breaker succeed on unknown mem usage
With this commit we implement a workaround for
https://bugs.openjdk.java.net/browse/JDK-8207200 which is a race
condition in the JVM that results in `IllegalArgumentException` to be
thrown in rare cases when we determine memory usage via `MemoryMXBean`.
As we do not want to fail requests in those cases we always return zero
memory usage.

Relates #31767
Relates #33125
2018-08-27 07:09:27 +02:00
Jason Tedor 143cd9bbaa
Do not lose default mapper on metadata updates (#33153)
When applying index metadata updates we run through the mappings
updating them if needed. Today if there is not an update to the default
mapper, we can lose the default mapping. This means that, for example,
if we apply a settings update to an index we will lose the default
mapper. This happens because we were not guarding updating the default
mapping with a check that the default mapping was updated in the
metadata update. When there is no update in the metadata update, we need
to continue to preserve the previous default mapping. This commit
achieves this by moving the updating of the default mapping under the
same guard that we use for updating the default mapping source. We add a
test that fails before putting the update under a guard and now passes
after moving the update under the guard.
2018-08-26 15:57:52 -04:00
Jason Tedor f8b07a0d84
Fix a mappings update test (#33146)
This commit fixes a mappings update test. The test is broken in the
sense that it passes, but for the wrong reason. The test here is testing
that if we make a mapping update but do not commit that mapping update
then the mapper service still maintains the previous document
mapper. This was not the case long, long ago when a mapping update would
update the in-memory state before the cluster state update was
committed. This test was passing, but it was passing because the mapping
update was never even updated. It was never even updated because it was
encountering a null pointer exception. Of course the in-memory state is
not going to be updated in that case, we are simply going to end up with
a failed cluster state update. Fixing that leads to another issue which
is that the mapping source does not even parse so again we would, of
course, end up with the in-memory state not being modified. We fix these
issues, assert that the result cluster state task completed
successfully, and finally that the in-memory state was not updated since
we never committed the resulting cluster state.
2018-08-26 09:36:17 -04:00
Albert Zaharovits fbe609d589
Reload Secure Settings REST specs & docs (#32990)
This is a minimal REST API spec and docs for the REST handler
for the `_nodes/reload_secure_settings endpoint`.

Relates #29135
2018-08-26 14:49:32 +03:00
Albert Zaharovits c567ec4a0f
Refactor CachingUsernamePassword realm (#32646)
Refactors the logic of authentication and lookup caching in
`CachingUsernamePasswordRealm`. Nothing changed about
the single-inflight-request or positive caching.
2018-08-26 14:09:23 +03:00
Simon Willnauer 3376922e8b
Add proxy support to RemoteClusterConnection (#33062)
This adds support for connecting to a remote cluster through
a tcp proxy. A remote cluster can configured with an additional
`search.remote.$clustername.proxy` setting. This proxy will be used
to connect to remote nodes for every node connection established.
We still try to sniff the remote clsuter and connect to nodes directly
through the proxy which has to support some kind of routing to these nodes.
Yet, this routing mechanism requires the handshake request to include some
kind of information where to route to which is not yet implemented. The effort
to use the hostname and an optional node attribute for routing is tracked
in #32517

Closes #31840
2018-08-25 20:41:32 +02:00
Nhat Nguyen 9dad82ece8
TEST: Skip assertSeqNos for closed shards (#33130)
If a shard was closed, we return null for SeqNoStats. Therefore the
assertion assertSeqNos will hit NPE when it verifies a closed shard.

This commit skips closed shards in assertSeqNos and enables this
assertion in AbstractDisruptionTestCase.
2018-08-24 21:02:13 -04:00
Nhat Nguyen 739a8d3d44
TEST: resync operation on replica should acquire shard permit (#33103)
This change makes sure that resync operations on replicas in the test
framework are executed under shard permits as the production code.
2018-08-24 20:25:13 -04:00
Nik Everett 1e9144d8e6
Switch remaining x-pack tests to new style Requests (#33108)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/qa/saml-idp-tests` and
`x-pack/qa/security-setup-password-tests` projects to use the new
versions.
2018-08-24 16:39:08 -04:00
Nik Everett dd4a8dc444
Switch remaining tests to new style Requests (#33109)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `client` and `distribution` projects to use
the new versions.
2018-08-24 16:37:25 -04:00
Nik Everett 8bee6b3a92
Switch remaining ml tests to new style Requests (#33107)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/plugin/ml/qa/native-multi-node-tests`,
`x-pack/plugin/ml/qa/single-node-tests` projects to use the new
versions.
2018-08-24 16:36:40 -04:00
Nik Everett a9a66a09dc Build: Line up IDE detection logic
The IDE detection logic in build.gradle and settings.gradle has to match
or else Eclipse users will have a bad time. But in this case I think it
mostly just hits folks who use the Eclipse compiler server like me.
2018-08-24 16:11:07 -04:00
Jay Modi b86dad22ce
Security index expands to a single replica (#33131)
This change removes the use of 0-all for auto expand replicas for the
security index. The use of 0-all causes some unexpected behavior with
certain allocation settings. This change allows us to avoid these with
a default install. If necessary, the number of replicas can be tuned by
the user.

Closes #29933
Closes #29712
2018-08-24 12:51:22 -06:00
Benjamin Trent 52cf57ee2d
HLRC: request/response homogeneity and JavaDoc improvements (#33133) 2018-08-24 13:18:50 -05:00
Nik Everett a023e64801 Checkstyle!
Catching your unused imports since 2001.
2018-08-24 14:13:13 -04:00
Jim Ferenczi 70030c18f1 [Test] Fix sporadic failure in MembershipActionTests
Rewrite test that require Version.V_5 constants.
2018-08-24 18:40:04 +02:00
Mayya Sharipova 6f1ee76443 Revert "Do NOT allow termvectors on nested fields (#32728)"
This reverts commit fdff8f3db0.
2018-08-24 10:12:16 -04:00
Tanguy Leroux 7e5efad929
[Rollup] Move toAggCap() methods out of rollup config objects (#32583) 2018-08-24 15:31:41 +02:00