4375 Commits

Author SHA1 Message Date
Igor Motov
7597fa8ae9 Use java.nio.charset.StandardCharsets instead of com.google.common.base.Charsets
We are trying to get rid of guava, so it doesn't make much sense to use it as a dependency unless we really need to.

Original commit: elastic/x-pack-elasticsearch@e2e2d58fba
2016-12-29 10:49:40 -05:00
Tim Vernum
22a32c8748 Add logstash_system user to X-Pack security reserved realm
The `logstash_system` user/role are used by logstash internals (e.g. monitoring), and work similarly to the Kibana user and role

Included features:  
- New logstash_system user/role (elastic/elasticsearch#4350, elastic/elasticsearch#4021)
- We no longer hide the reserved roles in the API (applies to both kibana and logstash_system)
- Automatically Disable logstash_system user on upgrade. If upgrading from a pre-5.2.0 version, automatically set the `logstash_system` user to disabled so that the system stays locked down. (elastic/elasticsearch#4445, elastic/elasticsearch#4394)

Closes: elastic/elasticsearch#4021
Closes: elastic/elasticsearch#4394 

Original commit: elastic/x-pack-elasticsearch@a9e6d1f216
2016-12-29 12:36:47 +11:00
Igor Motov
576a512ef2 Remove PROTO-based custom cluster state components
Switches custom cluster state components from PROTO-based de-serialization to named objects based de-serialization

Original commit: elastic/x-pack-elasticsearch@e1be8597ee
2016-12-28 12:38:03 -05:00
Yannick Welsch
505ed14c97 Separate cluster update tasks that are published from those that are not
Companion commit to elastic/elasticsearchelastic/elasticsearch#21912

Original commit: elastic/x-pack-elasticsearch@4ba569f8da
2016-12-23 12:21:59 +01:00
Jason Tedor
935eec54c5 Enable assertions in integration tests
This commit fixes some bugs that were found after enabling assertions in
integration tests.

Relates elastic/elasticsearch#4443

Original commit: elastic/x-pack-elasticsearch@4fdd84166d
2016-12-22 20:14:02 -05:00
Nik Everett
4ff6279865 Remove exception from client suite
Original commit: elastic/x-pack-elasticsearch@51e94561d9
2016-12-22 09:10:42 -05:00
Areek Zillur
84db1b8731 x-pack changes for elasticsearchelastic/elasticsearch#21964
In https://github.com/elastic/elasticsearch/pull/21964, index
and delete operations are executed as single item bulk requests
internally. This means index and delete operations use the
bulk transport endpoints (indices:data/write/bulk[s][p] and
indices:data/write/bulk[s][r]).

This PR adds bulk transport endpoint to 'write' and 'delete'
index privilages and adds index and delete action as composite
actions to delay the authentication to the shard level.

Original commit: elastic/x-pack-elasticsearch@2305fc9ca0
2016-12-22 02:35:35 -05:00
Jason Tedor
d5db68702c Adapt to method name change in core
This commit adapts to the change of the method name
ESTestCase#randomPostivieLong to ESTestCase#randomNonNegativeLong.

Original commit: elastic/x-pack-elasticsearch@689429cb54
2016-12-21 23:19:52 -05:00
jaymode
880808c428 test: disable security manager for custom realm integration tests
Original commit: elastic/x-pack-elasticsearch@4bcd7675b2
2016-12-21 16:05:43 -05:00
jaymode
9288401767 test: do not use security manager for client tests
Original commit: elastic/x-pack-elasticsearch@1200a1b757
2016-12-21 14:46:18 -05:00
Daniel Mitterdorfer
b973d10685 Improve BWC
Original commit: elastic/x-pack-elasticsearch@f84cb5b1fa
2016-12-21 16:43:55 +01:00
Nik Everett
483711ed20 Handle removal of IndiciesQueriesRegistry
It has been replaced by `XContentParser#namedObject`.

Original commit: elastic/x-pack-elasticsearch@fd316791a2
2016-12-21 08:01:49 -05:00
Daniel Mitterdorfer
fa1219fd13 Merge remote-tracking branch 'origin/master' into strict-booleans
Original commit: elastic/x-pack-elasticsearch@6fbbaf19dd
2016-12-21 08:47:45 +01:00
Jason Tedor
1178112c4a Skip pre-built transport client test on JDK 9
This test will currently not pass because Netty tries to do unsafe
operations that JDK 9 shoots down.

Relates elastic/elasticsearch#4430

Original commit: elastic/x-pack-elasticsearch@42aa250d4c
2016-12-20 20:25:41 -05:00
Tim B
f45c24faf2 Fix x-pack readme to reflect project name and dev-setup changes (elastic/elasticsearch#4402)
* Update readme to reflect new dev setup directory structure.

* Fix typo in elasticsearch-extra path in readme

* Update gradle exception for x-pack directory structure.

* Make directory path where x-pack must be checked out explicit in the gradle exception

Original commit: elastic/x-pack-elasticsearch@91f1d04542
2016-12-20 12:43:32 -06:00
Nik Everett
b2dc1bdca3 Introduce NamedXContentRegistry (elastic/elasticsearch#4399)
This "super registry" will eventually replace things like
`IndiciesQueriesRegistry` but for now it is just another thing
to plumb across requests.

Original commit: elastic/x-pack-elasticsearch@da26a42b36
2016-12-20 11:05:42 -05:00
Alexander Reelsen
6f7a065605 Watcher: Parse index action results to prevent exceptions (elastic/elasticsearch#4424)
When the index action is used to do some bulk indexing, the single
items of the response were not checked to have been indexed successful.
This could lead to NPEs due to an index response being null when the index
operation had failed. The action was still logged as a success though.

This commit only returns SUCCESS for the action, if all items were indexed
successfully. If all items failed, the result will be FAILED as well. Lastly
the result status PARTIAL_FAILURE is used if there were successful and unsuccessful
index operations.

Additionally some minor cleanups happened, like changing equals/hashcode.

Closes elastic/elasticsearch#4416

Original commit: elastic/x-pack-elasticsearch@692687e1af
2016-12-20 16:53:12 +01:00
Dimitrios Liappis
17c3164979 [ci] Fix bug with -Dtests.jvm.argline
The special `-Dtests.jvm.argline` params needed for jdk-9 builds do not
get passed correctly if enclosed within single quotes.

Fix jdk9 target for `dev-tools/ci` script to correctly pass the
-Dtests.jvm.argline parameters.

Relates:  elastic/elasticsearch#4428 

Original commit: elastic/x-pack-elasticsearch@6cd329b8da
2016-12-20 16:35:18 +02:00
Alexander Reelsen
0f8e6354f6 Watcher: Allow deletion of watches regardless of license (elastic/elasticsearch#4414)
The user might want to delete watches on expiry to prevent
further execution, without any actions being executed.

Closes elastic/elasticsearch#4413

Original commit: elastic/x-pack-elasticsearch@244a1b9cd8
2016-12-20 15:26:06 +01:00
Yannick Welsch
68bfc90cea Let ClusterStateObserver only hold onto state that's needed for change detection
Companion commit for elastic/elasticsearchelastic/elasticsearch#21631

Original commit: elastic/x-pack-elasticsearch@3333480707
2016-12-20 15:23:00 +01:00
Alexander Reelsen
b994f16f4a Watcher: Ensure non duplicate Wid ids (elastic/elasticsearch#4423)
Removing the WatchLockService could result in duplication of wids, because of a wrong
call to replace underscores with dashes. As UUIDs.createBase64UUID() can contain underscores
but they are kind of reserved in the Wid class due to handling of watch ids, this just uses
the toString() representation of a random UUID.

Closes elastic/elasticsearch#4422

Original commit: elastic/x-pack-elasticsearch@dceb01ae5e
2016-12-20 14:02:00 +01:00
Daniel Mitterdorfer
b9a38d9b97 Enable trace logging for LocalExporterTests#testSimpleExport()
This test fails due to a changing cluster state(?).  The
test checks that a local exporter is ready and then continues.
However, during the test, we see output similar to:

skipping exporter [_local] as it isn't ready yet

Which indicates that the cluster state has changed and the
exporter does not return a bulk anymore. Hence, the test is
failing although at one point in time it returned a bulk.

By enabling trace logging we should be able to find out
what's going on.

Original commit: elastic/x-pack-elasticsearch@d7e2200dd9
2016-12-20 08:54:26 +01:00
Ryan Ernst
789df7d2fa Internal: Update signatures for EnvironmentAwareCommand (elastic/elasticsearch#4367)
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#22175

Original commit: elastic/x-pack-elasticsearch@4359cb1947
2016-12-19 15:23:53 -08:00
javanna
adf22c3d24 Adapt to strict parsing mode removal
Original commit: elastic/x-pack-elasticsearch@2d6276a60f
2016-12-19 19:39:08 +01:00
Dimitrios Liappis
8a3181ceed [ci] Add jdk9 target to the dev-tools/ci script
To add support for jdk9 tests for x-pack we need a number of parameters
for the jvm, passed in the gradle command. To resolve issues with arg
passing through bash variables and improve readability, switch to an
array construct for storing the arguments.

Add new jdk9 target to allow execution of jdk-9-ea based tests in the CI
infra.

Relates: elastic/elasticsearch#4415 
Relates: https://github.com/elastic/infra/issues/1573#issuecomment-267832307

Original commit: elastic/x-pack-elasticsearch@b41e5f4b98
2016-12-19 19:53:00 +02:00
Alexander Reelsen
79a8f27569 Watcher: Remove WatchLockService (elastic/elasticsearch#4365)
The watch lock service is not really needed, as there is already
a data structure that has information about the currently executing
watches, that can be consulted before executed.

This change will now check, if there is already a watch running with
the current id. If there is not, execution will happen as usual. If
there is however, than a watch record will be created, stating that
the watch is currently being executed - which means that it is either
being executed or in the list of planned executions.

This way users can check in the watch history, if a watch has been executed
more often than it should.

In order to easily search for this, a new execution state called
`NOT_EXECUTED_ALREADY_QUEUED` has been added.

Original commit: elastic/x-pack-elasticsearch@867acec3c3
2016-12-19 17:33:48 +01:00
Daniel Mitterdorfer
2cb41e690d Address initial review comments
Original commit: elastic/x-pack-elasticsearch@bcd6433afb
2016-12-19 13:27:40 +01:00
Daniel Mitterdorfer
24a4290379 Merge remote-tracking branch 'origin/master' into strict-booleans
Original commit: elastic/x-pack-elasticsearch@32a533f38f
2016-12-19 12:58:30 +01:00
Daniel Mitterdorfer
7a652fa090 Merge pull request elastic/elasticsearch#4398 from danielmitterdorfer/xcontent_strict_duplicate_checks
Enable strict duplicate checks for all XContent types

Original commit: elastic/x-pack-elasticsearch@ad249c2e65
2016-12-19 09:30:19 +01:00
Tim Vernum
df7c528c66 Change classification of event_type in index audit trail (elastic/elasticsearch#4375)
In LoggingAuditTrail and the audit-event filtering, we distinguish between single-realm authentication failures ("realm_authentication_failure") and global failures ("authentication_failure").

Update the message output of IndexAuditTrail to reflect this distinction.

Original commit: elastic/x-pack-elasticsearch@56802ae2df
2016-12-19 14:39:21 +11:00
Simon Willnauer
c9cfedc3cb [TEST] check if .watches index / alias exists before creating it
Original commit: elastic/x-pack-elasticsearch@74d85121eb
2016-12-18 10:17:43 +01:00
Areek Zillur
278496d5e5 x-pack changes for removing deprecated _suggest endpoint (elasticsearchelastic/elasticsearch#22203)
Original commit: elastic/x-pack-elasticsearch@b9bba20fae
2016-12-16 13:06:35 -05:00
Daniel Mitterdorfer
d3c1fbe002 Enable strict duplicate checks for all XContent types
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default for all XContent types (not only JSON).

We have also changed the name of the system property to disable this feature
from `es.json.strict_duplicate_detection` to the now more appropriate name
`es.xcontent.strict_duplicate_detection`.

Relates elastic/elasticsearchelastic/elasticsearch#19614
Relates elastic/elasticsearchelastic/elasticsearch#22073

Original commit: elastic/x-pack-elasticsearch@c92e2a6c55
2016-12-16 13:57:55 +01:00
Alexander Reelsen
0d9cb5f7bd Docs: Updating execution phases of watch execution
Closes elastic/elasticsearch#4079

Original commit: elastic/x-pack-elasticsearch@658ba0a7f7
2016-12-16 11:30:27 +01:00
Daniel Mitterdorfer
e3302da589 Make boolean conversion strict
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.

Original commit: elastic/x-pack-elasticsearch@6400f18911
2016-12-15 19:06:59 +01:00
Boaz Leskes
b2972a142c Remove Cluster State Status (elastic/elasticsearch#4313)
A companion PR for https://github.com/elastic/elasticsearch/pull/21817

Original commit: elastic/x-pack-elasticsearch@392235877f
2016-12-15 17:07:02 +01:00
Alexander Reelsen
e6ee905931 Watcher: Remove unused class
Original commit: elastic/x-pack-elasticsearch@ecd48b7914
2016-12-15 11:33:36 +01:00
Ryan Ernst
6d4d599f91 Build: Convert xplugins to use new extra projects setup (elastic/elasticsearch#4175)
* Build: Convert xplugins to use new extra projects setup

This change makes the gradle initialization for xplugins look in the
correct location for elasticsearch, which is now as a sibling of an
elasticsearch-extra directory, with x-plugins as a child of the extra
directory.

The elasticsearch side of this change is
elastic/elasticsearchelastic/elasticsearch#21773. This change will enable renaming x-plugins
to x-pack, see elastic/elasticsearch#3643.

Original commit: elastic/x-pack-elasticsearch@09398aea5a
2016-12-14 15:02:28 -08:00
Nik Everett
597ca9275e Finish centralizing XContentParser creation in tests
Now we're ready to add `NamedXContentRegistry`.

Original commit: elastic/x-pack-elasticsearch@c4f6c7c3d6
2016-12-14 11:19:09 -05:00
Daniel Mitterdorfer
911aec0502 Enable strict duplicate checks for JSON content
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default. This ensures that JSON keys are always unique. While this has
a performance impact, benchmarking has indicated that the typical drop in
indexing throughput is around 1 - 2%.

As a last resort, we allow users to still disable strict duplicate checks
by setting `-Des.json.strict_duplicate_detection=false` which is
intentionally undocumented.

Relates elastic/elasticsearchelastic/elasticsearch#19614
Relates elastic/elasticsearchelastic/elasticsearch#22073

Original commit: elastic/x-pack-elasticsearch@47055336e3
2016-12-14 10:29:21 +01:00
Daniel Mitterdorfer
feabaaf1d6 Add missing import
Original commit: elastic/x-pack-elasticsearch@fbc8eb7ac6
2016-12-14 09:34:55 +01:00
Daniel Mitterdorfer
83240e25b4 Merge remote-tracking branch 'origin/master' into json_strict_duplicate_checks
Original commit: elastic/x-pack-elasticsearch@af25e460d0
2016-12-14 08:53:45 +01:00
Nik Everett
d4939e1972 Start centralizing XContentParser creation in tests
Original commit: elastic/x-pack-elasticsearch@65be7ecd27
2016-12-13 20:28:19 -05:00
Spencer
c3d8fd4191 Merge pull request elastic/elasticsearch#3259 from spalger/upgrade/eslint
[ui] Upgrade eslint

Original commit: elastic/x-pack-elasticsearch@2e47b105ff
2016-12-13 17:39:25 -07:00
Jason Tedor
3f5d24ae55 Add shutdown hook for closing CLI commands
This commit adapts some CLI command tests for some changes in core.

Relates elastic/elasticsearch#4342

Original commit: elastic/x-pack-elasticsearch@1e48c1404f
2016-12-13 19:10:35 -05:00
Alexander Reelsen
1e7985563a Tests: Make MockWebServer thread-safe (elastic/elasticsearch#4351)
The new MockWebServer assumed that responses are inserted at the beginning
and removed later on. This was not thread safe.

Also this fixes a bug in the HttpExporterIT where there was no wait time
for a bulk request, even though the request execution is asynchronous.

Closes elastic/elasticsearch#4335

Original commit: elastic/x-pack-elasticsearch@11f31f68bd
2016-12-13 19:34:33 +01:00
Daniel Mitterdorfer
8be3250601 Merge remote-tracking branch 'origin/master' into json_strict_duplicate_checks
Original commit: elastic/x-pack-elasticsearch@a010b1b4e9
2016-12-13 16:22:47 +01:00
Daniel Mitterdorfer
7cb0771c46 Use assume* instead of custom skipping logic
Original commit: elastic/x-pack-elasticsearch@0f3a01735f
2016-12-13 16:21:46 +01:00
Luca Cavanna
1c846dd893 Stop execution once destructive operations check has failed (elastic/elasticsearch#4337)
Otherwise we do return an error, but we also go ahead and open/close/delete the indices anyways.

Original commit: elastic/x-pack-elasticsearch@176eca4cff
2016-12-13 11:22:48 +01:00
Alexander Reelsen
b57c4f6ebe Watcher: Remove in memory watch store (elastic/elasticsearch#4201)
In order to prepare to the distributed watch execution, this commit
removes the in memory watch store.

Whenever a watch is needed now, a get request is executed and the parsing
is done. This happens when

* Put
* Get
* Ack
* Activate/Deactivate
* Execute

Note: This also means there are no usage stats currently regarding
the watch count, because we would need to execute a query. This would
require the usage stats to be async, see elastic/elasticsearch#3569

Another advantage is, that there is no dirty flag in the watch itself
needed anymore, because the watch is always the latest. Also write
operations store immediately and dont leave anything in memory.

Also ActionListener.wrap() was used a lot instead of more verbose anonmyous
inner classes.

Original commit: elastic/x-pack-elasticsearch@c47465b47c
2016-12-13 08:54:03 +01:00