Commit Graph

1662 Commits

Author SHA1 Message Date
Alexander Reelsen d065b087ee Revert "Cleanup: Remove HaltedClock (elastic/x-pack-elasticsearch#3664)"
This reverts commit elastic/x-pack-elasticsearch@f91c401a60 due to
failing tests, like

./gradlew :x-pack-elasticsearch:plugin:watcher:test -Dtests.seed=AE30350FCE96D26D -Dtests.class=org.elasticsearch.xpack.watcher.watch.WatchTests -Dtests.method="testParserSelfGenerated" -Dtests.security.manager=true -Dtests.locale=ja-JP -Dtests.timezone=EET

Original commit: elastic/x-pack-elasticsearch@e45d79d643
2018-01-23 18:08:54 +01:00
David Roberts 697a08e742 [ML] Refactor so ML doesn't require PersistentTasksService at startup (elastic/x-pack-elasticsearch#3682)
At present the PersistentTasksService is created inside the ML plugin.
This is undesirable, as other plugins will use persistent tasks in the
near future.

This change refactors the startup code so that the PersistentTasksService
no longer needs to be passed to any constructors for ML components.

A future change will still be required to actually move the initialization
of the PersistentTasksClusterService, PersistentTasksService and
PersistentTasksExecutorRegistry out of the ML plugin, but following this
change it should be fairly simple.

Original commit: elastic/x-pack-elasticsearch@3c2a8e020e
2018-01-23 16:55:08 +00:00
Simon Willnauer 63c0e288af Ensure we protect Collections obtained from scripts from self-referencing (elastic/x-pack-elasticsearch#3681)
Self referencing maps can cause SOE if they are iterated ie. in their toString methods. This chance adds some protected to the usage of those collections.
see elastic/elasticsearch#28335

Original commit: elastic/x-pack-elasticsearch@c4f1089c74
2018-01-23 17:02:37 +01:00
Dimitris Athanasiou 215f9af1cc [ML] Add trace logging for when search response is obtained (elastic/x-pack-elasticsearch#3669)
This is useful for understanding performance characteristics
as it helps us understand whether the bottleneck is the search
part or the analytics part.

Relates elastic/x-pack-elasticsearch#3590

Original commit: elastic/x-pack-elasticsearch@dc8c095958
2018-01-23 14:16:34 +00:00
Alexander Reelsen e3da8fa4ae Cleanup: Remove HaltedClock (elastic/x-pack-elasticsearch#3664)
The HaltedClock was a leftover from moving over from our own Clock
implementation to a java.time one. java.time already has a fixed clock,
this one is not needed.

Original commit: elastic/x-pack-elasticsearch@f91c401a60
2018-01-23 14:13:22 +01:00
Simon Willnauer c70c1c7e98 Expose XPackExtensions via SPI (elastic/x-pack-elasticsearch#3530)
This change adds SPI loading for XPackExtensions that allows to extend
XPack via an ordinary plugin. This can co-exist with the existin
extension mechanism for the time being.

Original commit: elastic/x-pack-elasticsearch@bf02b56dee
2018-01-23 13:05:39 +01:00
David Kyle 8f393c3066 [ML] Improve error message when creating calendars (elastic/x-pack-elasticsearch#3668)
Original commit: elastic/x-pack-elasticsearch@996b0e2f65
2018-01-23 09:34:07 +00:00
Simon Willnauer c3efa4b6bc [TEST] disable ML when ML is not installed
Original commit: elastic/x-pack-elasticsearch@cd84acc3e0
2018-01-23 10:28:37 +01:00
Ryan Ernst 9d87b63ca4 Build: Fix third party audit task for xpack core (elastic/x-pack-elasticsearch#3656)
This commit re-enables thirdPartyAudit for x-pack core. Previously, when
xpack was a single plugin, it transitively picked up httpcore-nio
through the elasticsearch rest client. Now that xpack core does not
depend on the rest client, httpcore-nio must be added as a dependency.
Additionally, commons-logging was previously handled through the rest
client, but now xpack depends directly on this, thus excludes must be
added for the pesky missing classes there.

This commit also cleans up unnecessary parts of plugin/build.gradle no
longer necessary.

Original commit: elastic/x-pack-elasticsearch@70e936bdc3
2018-01-22 22:58:34 -08:00
jaymode 698ad62150 Fix SAMLRealmTests#testReadIdpMetadataFromHttps failures
This commit fixes failures in SAMLRealmTests#testReadIdpMetadataFromHttps. The main test failure is
caused by the test relying too much on timing with a sleep; the sleep has been removed and replaced
with an assert busy. The second issue that was fixed in the test, is the test makes two requests to
the webserver but only one request is queued in the MockWebServer leading to a NPE.

relates elastic/x-pack-elasticsearch#3667

Original commit: elastic/x-pack-elasticsearch@6c7076c990
2018-01-22 12:20:52 -07:00
Jay Modi e577d2b776 Fix handling of IOException when computing key for tokens (elastic/x-pack-elasticsearch#3639)
The invalid token test has been failing due to the way the test expects an exception to be
returned. Recent changes allowed the exception to be returned via a listener but the test
was not adapted and the code in the token service was not properly handling these exceptions
when they were returned via a listener.

relates elastic/x-pack-elasticsearch#3630

Original commit: elastic/x-pack-elasticsearch@1cf2cc0427
2018-01-22 09:09:20 -07:00
Jason Tedor 2453da82e0 Force evaluation order between sub-plugins and core
We are relying on the fact that :x-pack-elasticsearch:plugin:core is
configured before these. This happens when these projects are
alphabetically after core, but is not the case when they are not
(ccr). To address this, we simplfy be explicit about evaluation order in
all sub-plugins.

Relates elastic/x-pack-elasticsearch#3663

Original commit: elastic/x-pack-elasticsearch@755f84258b
2018-01-22 10:55:03 -05:00
Jason Tedor 520b372549 Remove watcher as a security test dependency
This commit removes Watcher as a test dependency for Security.

Relates elastic/x-pack-elasticsearch#3655

Original commit: elastic/x-pack-elasticsearch@8d770ffb9d
2018-01-22 10:16:03 -05:00
Luca Cavanna d9eb4c049f Trim down usages of `ShardOperationFailedException` interface (elastic/x-pack-elasticsearch#3662)
In many cases we use the `ShardOperationFailedException` interface to abstract an exception that can only be of one type, namely `DefaultShardOperationException`. There is no need to use the interface in such cases, the concrete type should be used instead. That has the additional advantage of simplifying parsing such exceptions back from rest responses for the high-level REST client.

Original commit: elastic/x-pack-elasticsearch@b2259afcbf
2018-01-22 15:51:55 +01:00
Dimitrios Athanasiou 8ee5b608e0 [ML] Fix line width violation
Original commit: elastic/x-pack-elasticsearch@94d55c42b8
2018-01-22 14:17:42 +00:00
Dimitris Athanasiou cc432aee34 [ML] Check calendar exists before removing job (elastic/x-pack-elasticsearch#3661)
Also, removes check for whether a job-to-remove exists
and replaces it with a check of whether a job-to-remove
is already present in the calendar. This allows to
remove a job that may no longer exists and it improves
feedback for the case that an existing job is removed from
a calendar that doesn't contain it.

relates elastic/x-pack-elasticsearch#3620

Original commit: elastic/x-pack-elasticsearch@3ea39be1b6
2018-01-22 13:49:38 +00:00
Simon Willnauer 129f021843 [TEST] Fix SettingsFilterTests to also install the Security plugin
Original commit: elastic/x-pack-elasticsearch@b30997dc6a
2018-01-22 14:27:42 +01:00
Dimitris Athanasiou f3282b559f [ML] Check calendar exists before deleting event (elastic/x-pack-elasticsearch#3659)
Solves the event part of elastic/x-pack-elasticsearch#3620

Original commit: elastic/x-pack-elasticsearch@c79ca85c6e
2018-01-22 13:02:42 +00:00
Ryan Ernst ac44ee8257 Build: Move test files from plugin/src to appropriate other projects (elastic/x-pack-elasticsearch#3653)
This commit moves mostly security tests that were left in plugin/src. It
also moves the CompositeTestingXPackPlugin into tribe license tests qa
project, which was the only remaining use. This class needs to be
removed, as it has unavoidable problems with dependency conflicts (ie
guava) between security and watcher, which it pulls in both into the
test classpath.

Original commit: elastic/x-pack-elasticsearch@756209e010
2018-01-21 14:08:47 -05:00
Jason Tedor 4d4f979d61 Remove debugging println from API JAR task
This commit removes a leftover println that was added while debugging
the execution of the API JAR task.

Original commit: elastic/x-pack-elasticsearch@37c2e8fe5b
2018-01-21 13:20:03 -05:00
Jason Tedor 04b48324aa Drop native controller from descriptors (except ML)
These were copied wholesale from the pre-split X-Pack
descriptor. However, only ML has a native controller. This removes the
plugin installation asking multiple times to approve the existence of a
native controlled for every bundled plugin.

Relates elastic/x-pack-elasticsearch#3650

Original commit: elastic/x-pack-elasticsearch@4fca606243
2018-01-21 11:00:51 -05:00
Jason Tedor a582acddf8 Add API JAR task to x-pack-core
This commit adds back the task for assembling the API JAR from the
x-pack-core JAR.

Original commit: elastic/x-pack-elasticsearch@10385ecf17
2018-01-21 10:00:47 -05:00
Tim Vernum 47213f5675 [Security] Add SAML authentication support (elastic/x-pack-elasticsearch#3646)
Introduces:

- SAML Realm
- REST & Transport actions to support SAML single signon / signout
- Tests for above
- More XML than you ever wanted to see.

Original commit: elastic/x-pack-elasticsearch@b0fe7bb652
2018-01-21 08:43:00 +10:00
Jason Tedor 50864eabce Introduce plugin-specific env scripts
With plugins of a meta-plugin now loaded in separate classloaders, we
should not be loading all classes in all plugins of a meta-plugin when
executing scripts. This is particularly problematic in the case of
security extensions where the install plugin extension command would be
running with the classpath of all plugins. However, if there is JAR hell
in this classpath, installation would fail. This is not realistic though
since the plugins are run in separate classloaders. To fix this, for the
scripts of a plugin, we only set the classpath to include the JARs for
that plugin and the JARs of core. This leads us to the introduction of
plugin-specific env scripts.

Relates elastic/x-pack-elasticsearch#3649

Original commit: elastic/x-pack-elasticsearch@543df37eed
2018-01-20 15:56:00 -05:00
Jason Tedor 51c53710d7 Keep imports consistent with 6.x
Some imports were changed in 6.x to address line-length issues
there. This commit pulls the same changes to master to keep the branches
consistent to simplify backports.

Original commit: elastic/x-pack-elasticsearch@190f9d41f5
2018-01-20 14:52:07 -05:00
David Roberts 64cfa017f0 [TEST] Update ML categorization expected result following C++ change
Original commit: elastic/x-pack-elasticsearch@cc78772aa2
2018-01-20 14:50:24 +00:00
Jason Tedor 60e577a9c4 Fix test base class that disables ML autodetect
This is no longer needed for general X-Pack since the tests will not be
depending on ML. We move this class to an ML specific directory and
remove the dependency from other tests.

Original commit: elastic/x-pack-elasticsearch@9b287f7460
2018-01-20 01:27:33 -05:00
Michael Basnight f3ec4a5208 Split up xpack plugins into their own modules (elastic/x-pack-elasticsearch#3643)
Thanks to some great work by a bunch of amazing people, the chuck norris xpack split is a go!

Original commit: elastic/x-pack-elasticsearch@dad98e28f4
2018-01-19 23:30:17 -06:00
Jason Tedor 0602d38bd5 Split transport implementations into client/server
This commit splits the transport implementations into components that
can be used client-side (in the transport client) and server-side (in
the server). This enables removing security as a dependency for the
transport client.

Relates elastic/x-pack-elasticsearch#3635

Original commit: elastic/x-pack-elasticsearch@e480eb7eb2
2018-01-19 12:09:34 -05:00
Dimitrios Athanasiou 9e834cbf8a [ML] Fix javadoc in MlStrings#hasValidLengthForId
Original commit: elastic/x-pack-elasticsearch@ef5ecad5e2
2018-01-19 11:45:41 +00:00
Dimitris Athanasiou 21f692c02b [ML] Further validate calendar_id and add calendar description (elastic/x-pack-elasticsearch#3624)
relates elastic/x-pack-elasticsearch#3595

Original commit: elastic/x-pack-elasticsearch@fade977361
2018-01-19 10:44:39 +00:00
Ryan Ernst 19874e35ee More license file corrections
See elastic/x-pack-elasticsearch@0af9ffc

Original commit: elastic/x-pack-elasticsearch@360a88381f
2018-01-18 16:34:25 -08:00
Ryan Ernst 43486d6a70 Remove unnecessary license files
These are no longer necessary after https://github.com/elastic/elasticsearch/pull/28304

Original commit: elastic/x-pack-elasticsearch@0af9ffc4d1
2018-01-18 16:19:10 -08:00
Tim Brooks 685b75da3a Support changes in nio channel contexts (elastic/x-pack-elasticsearch#3609)
This is related to elastic/elasticsearch#elastic/x-pack-elasticsearch#28275. It modifies x-pack to
support the changes in channel contexts. Additionally, it simplifies
the SSLChannelContext by relying on some common work between it and
BytesChannelContext.

Original commit: elastic/x-pack-elasticsearch@8a8fcce050
2018-01-18 13:06:42 -07:00
jaymode e775e84a7e Test: add additional logging to SslMultiPortTests
This commit adds some additional logging to the SslMultiPortTests to help with debugging an
intermittent failure.

Relates elastic/x-pack-elasticsearch#1809

Original commit: elastic/x-pack-elasticsearch@ae9c967249
2018-01-18 12:46:01 -07:00
jaymode 40d315b244 Test: only use a single shard in RemoteIndexAuditTrailStartingTests
Relates elastic/x-pack-elasticsearch#435

Original commit: elastic/x-pack-elasticsearch@05db61297b
2018-01-18 12:07:30 -07:00
Tim Brooks fb12a0e383 Support TLS/SSL renegotiation (elastic/x-pack-elasticsearch#3600)
This commit is related to elastic/x-pack-elasticsearch#3246. It adds support for receiving TLS/SSL
renegotiation requests for peers.

Original commit: elastic/x-pack-elasticsearch@c22c16b3bc
2018-01-18 10:59:44 -07:00
Dimitris Athanasiou eefd8e7940 [ML] Allow categorical conditions to use all analysis fields (elastic/x-pack-elasticsearch#3615)
Categorical conditions should be allowed to use any of the analysis
fields as a field_name.

Original commit: elastic/x-pack-elasticsearch@a0d25e7445
2018-01-18 15:48:56 +00:00
Ryan Ernst b785f9c61b Build: Use md5 to determine whether ml snapshot needs downloading (elastic/x-pack-elasticsearch#3612)
This commit makes ML snapshot downloading happen less often. It does
that by first moving the download location to a directory outside the
destructive power of gradle clean, and then also uses the md5 of the zip
to compare to that found in s3. This allows us to do a cheap HEAD
request to find if the file has changed.

Original commit: elastic/x-pack-elasticsearch@cd8b00fd31
2018-01-17 22:12:23 -08:00
Jay Modi 60d4b7e53e Add the ability to refresh tokens obtained via the API (elastic/x-pack-elasticsearch#3468)
This commit adds the ability to refresh tokens that have been obtained by the API using a refresh
token. Refresh tokens are one time use tokens that are valid for 24 hours. The tokens may be used
to get a new access and refresh token if the refresh token has not been invalidated or
already refreshed.

relates elastic/x-pack-elasticsearch#2595

Original commit: elastic/x-pack-elasticsearch@23435eb815
2018-01-17 12:18:44 -07:00
Dimitris Athanasiou f91631664a [ML] Rename Condition member valueFilter -> value (elastic/x-pack-elasticsearch#3599)
Original commit: elastic/x-pack-elasticsearch@5ef8415ca5
2018-01-17 18:39:47 +00:00
Tim Brooks dda3a8dee0 Add TLS/SSL enabled SecurityNioTransport (elastic/x-pack-elasticsearch#3519)
This is related to elastic/x-pack-elasticsearch#3246. This commit adds a SSL/TLS layer to the nio
work implemented in the SSLChannelContext and SSLDriver classes.
This work is used to build up a SecurityNioTransport implementation.
This transport does yet offer feature parity with our normal security
transport. It mainly offers SSL/TLS security.

Original commit: elastic/x-pack-elasticsearch@d0e0484418
2018-01-17 09:44:31 -07:00
Alexander Reelsen ef2d2764a5 Watcher: Improve cluster state listener behaviour (elastic/x-pack-elasticsearch#3538)
The cluster state listener used by watch now have two additional checks.
First, when no master node exists in the cluster state, watcher will
stop and the indexing listener will not try to trigger any new watch.
Second, when there is a global cluster write level block, it would not
be possible to update the watches index or write into the watcher
history, so the listener can bail at that case as well.

In addition this also changes the log level from debug to info when
watcher is stopped. It turned out that there are zero insights when or
if watcher is stopped when normal logging is activated. This makes it
super hard for support to know when watcher is stopped or started at all
due to shards being moved around.

Original commit: elastic/x-pack-elasticsearch@5e9ce24380
2018-01-17 14:18:17 +01:00
Tim Vernum b0552e1c6e [Security] Handle cache expiry in token service (elastic/x-pack-elasticsearch#3565)
* [Security] Handle cache expiry in token service

The keyCache on TokenService.KeyAndCache has a 60 minute expiry.
If the token service was idle for more than 60 minutes, the current
key would be expired and it would then fail to generate user tokens.

Original commit: elastic/x-pack-elasticsearch@fd98130a27
2018-01-17 13:04:59 +10:00
Yogesh Gaikwad 29663c1f38 Fix for Issue elastic/x-pack-elasticsearch#3403 - Predictable ordering of security realms (elastic/x-pack-elasticsearch#3533)
* Security Realms: Predictable ordering for realms

To have predictable ordering of realms, by having secondary
sorting on realm name resulting in stable and consistent documentation.
Documentation update describing how ordering of realms is determined.
Testing done by adding unit test for the change, ran gradle clean check locally.

relates elastic/x-pack-elasticsearch#3403

Original commit: elastic/x-pack-elasticsearch@98c42a8c51
2018-01-17 10:29:00 +11:00
Jason Tedor 8a0a14b9a3 Adapt to compile and runtime Java home distinction
We now separate the compiler Java home from the time runtime Java home
(the one that is used to compile class files versus the one that is used
to run tests). This commit adapts x-pack-elasticsearch to this change.

Relates elastic/x-pack-elasticsearch#3477

Original commit: elastic/x-pack-elasticsearch@bdb096e21c
2018-01-16 13:45:42 -05:00
Dimitris Athanasiou 368c4fff56 [ML] Expand job groups on scheduled events update (elastic/x-pack-elasticsearch#3577)
Relates elastic/x-pack-elasticsearch#3016

Original commit: elastic/x-pack-elasticsearch@73334d8e01
2018-01-16 11:53:37 +00:00
Dimitrios Athanasiou 0b8723015f [ML] Capitalize first letter in error message
Original commit: elastic/x-pack-elasticsearch@7062851357
2018-01-16 11:50:39 +00:00
Dimitris Athanasiou 2350b85507 [ML] Improve error on get calendar when ml-meta index is missing (elastic/x-pack-elasticsearch#3576)
Relates elastic/x-pack-elasticsearch#3016

Original commit: elastic/x-pack-elasticsearch@bc60a0cb72
2018-01-16 11:00:07 +00:00
Alexander Reelsen ac2884f034 Watcher: Fix NPE in watcher index template registry (elastic/x-pack-elasticsearch#3571)
The current code throws an NPE, when there is no master node available
in the cluster state. This commit adds an additional check for an
existing master node.

relates elastic/x-pack-elasticsearch#3569

Original commit: elastic/x-pack-elasticsearch@e0024b5170
2018-01-16 10:11:12 +01:00