Commit Graph

7432 Commits

Author SHA1 Message Date
Tim Vernum 7bb2cba14f [Security] Reset IndexAuditTrail to INITIALISED before start (elastic/x-pack-elasticsearch#3807)
Calling start() when already in the STARTING state doesn't do anything, so the component
gets stuck in STARTING state forever.

Also: wait on the required index name not just the cluster.

Also: added more logging to help diagnose such issues (either in RemoteIndexAuditTrailStartingTests or production)

Original commit: elastic/x-pack-elasticsearch@fb81214fe7
2018-02-07 12:30:23 +11:00
jaymode 092fcd3675 Test: do not check cluster stats in audit qa project
This commit wraps the TestCluster in one that does not check stats to prevent tripping an
assertion for the accounting breaker as the index audit trail is running in the background and
will affect the value of the breakers.

See elastic/x-pack-elasticsearch#157

Original commit: elastic/x-pack-elasticsearch@381ebbd413
2018-02-06 13:35:09 -07:00
jaymode 4731a1b56b Test: disable monitoring for security spi qa project
This commit disables monitoring for the qa project that tests custom realms and role providers
as monitoring can cause failures due to an accounting breaker not being reset.

See elastic/x-pack-elasticsearch#157

Original commit: elastic/x-pack-elasticsearch@f882507e85
2018-02-06 13:09:09 -07:00
Jay Modi 18c04c714d [Security] handle null values when attempting to parse as DN when mapping roles (elastic/x-pack-elasticsearch#3791)
This commit adds special handling for null values when building a DistinguishedNamePredicate for
role mapping. Previously this would have resulted in an exception from the unboundid ldapsdk as
the DN is invalid.

relates elastic/x-pack-elasticsearch#3787

Original commit: elastic/x-pack-elasticsearch@9386dae03e
2018-02-06 10:19:49 -07:00
Nik Richers 20c1791dd8 Fix broken link for 6.2
Direct commit to speed up fix.

Original commit: elastic/x-pack-elasticsearch@f9187b81b8
2018-02-06 07:21:19 -08:00
Lee Hinman b2b95cb25d [TEST] Fix compilation for XContent parser
Original commit: elastic/x-pack-elasticsearch@bf03cc34fe
2018-02-05 16:56:36 -07:00
Tim Vernum 80b5ac9562 [DOC] SAML documentation (elastic/x-pack-elasticsearch#3657)
Includes:
- docs for new realm type "saml"
- docs for new settings for SAML realms
- a guide for setting up SAML accross ES + Kibana

Original commit: elastic/x-pack-elasticsearch@85f8f6d409
2018-02-05 12:22:54 +11:00
lcawley 66414afb30 [DOCS] Updated doc info in readme
Original commit: elastic/x-pack-elasticsearch@89b4ec79ba
2018-02-01 18:37:01 -08:00
Lisa Cawley 83910f717e [DOCS] Build docs from elasticsearch (elastic/x-pack-elasticsearch#3638)
Original commit: elastic/x-pack-elasticsearch@379e93a887
2018-02-01 13:35:42 -08:00
David Turner a1cb0911d3 Allow use of LocalDateTime#atZone() in DateTimeFormatterTimestampConverter (elastic/x-pack-elasticsearch#3811)
Adds a `@SuppressForbidden` annotation to permit use of a
trappy method from `java.time`, pending resolution of elastic/x-pack-elasticsearch#3810.

X-pack component of elasticsearch#28476.

Original commit: elastic/x-pack-elasticsearch@c5e1ae0970
2018-02-01 14:33:23 +00:00
Martijn van Groningen ee834026d1 Merge changes related to moving persistent tasks to elasticsearch repo into master branch.
Original commit: elastic/x-pack-elasticsearch@fb61e4853c
2018-02-01 15:31:57 +01:00
Dimitrios Athanasiou 05515b370a [ML][TEST] Increase timeout in MlDailyManagementServiceTests
relates elastic/x-pack-elasticsearch#2916

Original commit: elastic/x-pack-elasticsearch@2925dc45ea
2018-02-01 14:09:33 +00:00
Jason Tedor 2a7015a12c Bump Gradle to version 4.5
This commit bumps Gradle from version 4.3 to version 4.5.

Relates elastic/x-pack-elasticsearch#3805

Original commit: elastic/x-pack-elasticsearch@5e464a7d2b
2018-01-31 23:18:58 -05:00
Tim Vernum 415bb7f039 Clear Realm Caches on role mapping health change (elastic/x-pack-elasticsearch#3782)
If any of the follow take place on security index, then any cached role mappings
are potentially invalid and the associated realms need to clear any cached users.
- Index recovers from red
- Index is deleted
- Index becomes out-of-date / not-out-of-date

Original commit: elastic/x-pack-elasticsearch@1bcd86fcd4
2018-02-01 10:01:57 +11:00
Tal Levy a627fec53e fix timeout usage in _bulk to `timeout` from `master_timeout` (elastic/x-pack-elasticsearch#3796)
The HTTP Exporter in Monitoring allowed users to set a timeout parameters
for the requests. When set, this was setting the `master_timeout` query parameter
in Bulk Requests. The problem is that Bulk Requests do not support this type of
timeout.

Original commit: elastic/x-pack-elasticsearch@9be194006e
2018-01-31 09:13:46 -08:00
Martijn van Groningen d2e7cee41d fixed compile error
Original commit: elastic/x-pack-elasticsearch@ff02a3244c
2018-01-31 15:38:14 +01:00
Martijn van Groningen 5ed8d81df8 Disabled integtest in core, because there are no *IT tests anymore.
Original commit: elastic/x-pack-elasticsearch@b4343d3ec4
2018-01-31 15:10:43 +01:00
Alexander Reelsen c20f3ba996 Watcher: Add support for scheme in proxy configuration (elastic/x-pack-elasticsearch#3614)
This adds support to allow different schemes in a proxy being used
compared to what the actual request requires. So if your proxy runs via
HTTP, but the endpoint you want to connect to uses HTTPS, this is now
possible to configure the proxy explicitely.

Also a small unit test for parsing this has been added.

relates elastic/x-pack-elasticsearch#3596

Original commit: elastic/x-pack-elasticsearch@176f7cdf0e
2018-01-31 14:12:25 +01:00
Dimitris Athanasiou ed11dad855 [ML] Expand detector function shortcuts (elastic/x-pack-elasticsearch#3789)
relates elastic/x-pack-elasticsearch#3162


Original commit: elastic/x-pack-elasticsearch@a3512c6693
2018-01-31 13:11:32 +00:00
Alexander Reelsen 71f68d3413 Tests: Remove watcher ensureWachterStarted/Stopped methods (elastic/x-pack-elasticsearch#3798)
Tests have been failing because of out of order cluster state processing
or because of stopping/starting was still in progress. Current tests do
not do further stop/start tries after the first leading to potential
being stuck in an unwanted state. This commit removes the methods
checking for the state being started or stopped in favor of using the
stopWatcher/startWatcher methods which now check if the desired state is
reached and otherwise issue another start/stop command.

Original commit: elastic/x-pack-elasticsearch@97b3232a6a
2018-01-31 12:15:19 +01:00
Martijn van Groningen b2e0cfb111 Cut ml over to `PersistentTaskPlugin` and drop its own initialization of the persistent tasks framework.
Original commit: elastic/x-pack-elasticsearch@484b53c2a8
2018-01-31 11:08:45 +01:00
Martijn van Groningen 6f101d71d5 Removed persistent tasks framework from xpack-elasticsearch repository, because it now part of the elasticsearch repository.
Original commit: elastic/x-pack-elasticsearch@7a90ffc4f6
2018-01-31 11:05:54 +01:00
Tim Vernum d0f6d273d0 [TEST] Add additional logging to test
Attempt to diagnose elastic/x-pack-elasticsearch#3292 if it occurs again.

Original commit: elastic/x-pack-elasticsearch@eb262540f0
2018-01-31 18:26:57 +11:00
Nik Everett 99081adfbf XContent: Adapt to new method on parser (elastic/x-pack-elasticsearch#3797)
Properly delegates `XContentParser#getDeprecationHandler`.

Original commit: elastic/x-pack-elasticsearch@45356d600a
2018-01-30 18:21:21 -05:00
Simon Willnauer 570411c2dc Remove all tribe related code, comments and documentation (elastic/x-pack-elasticsearch#3784)
Relates to elastic/elasticsearch#28443

Original commit: elastic/x-pack-elasticsearch@5c4e7fccc7
2018-01-30 20:40:46 +01:00
Dimitris Athanasiou 5b7c38da7f [ML] UpdateProcessNotifier should drain the update queue (elastic/x-pack-elasticsearch#3774)
The notifier is scheduled to run once per second. Currently,
it simply polls for the next update in the queue. However,
when there are multiple updates queued up, there is no
reason to wait for subsequent runs in order to execute the
rest of the updates.

This commit changes the notifier to drain the queue each time
it runs. It then serially executes the updates.

relates elastic/x-pack-elasticsearch#3769

Original commit: elastic/x-pack-elasticsearch@7a433c17f2
2018-01-30 16:35:02 +00:00
David Roberts ce77a3dd6d [TEST] Make test more tolerant to slow VMs (elastic/x-pack-elasticsearch#3783)
The test this change applies to has failed a couple of times on
Windows VMs.

Original commit: elastic/x-pack-elasticsearch@94e94d3ffd
2018-01-30 16:34:16 +00:00
Andy Bristol e45b738b4c [TEST] remove check for extensions executables (elastic/x-pack-elasticsearch#3778)
These executables have been removed because the old plugin
infrastructure has been replaced with SPI

Follow up from elastic/x-pack-elasticsearch#3734

Original commit: elastic/x-pack-elasticsearch@89dd1ab3b1
2018-01-30 07:42:44 -08:00
Jay Modi 71788671ad Test: fix skip check for EllipticCurveSSLTests (elastic/x-pack-elasticsearch#3771)
This change fixes the skip check for the EllipticCurveSSLTests. The skip check that is in the test
was added to proctect against failures on JVMs that do not support EC ciphers such as the packaged
openjdk in some linux distributions. The old skip check did not execute until the cluster was up
so the test would still fail with errors such as no cipher suites in common. This change moves the
check into a before class method that checks availability.

Original commit: elastic/x-pack-elasticsearch@5550ca4e1e
2018-01-30 08:21:15 -07:00
jaymode 0f6ca4716b Test: increase time to wait for the tribe node to start
See elastic/x-pack-elasticsearch#3754

Original commit: elastic/x-pack-elasticsearch@b1fb53ce8e
2018-01-30 08:13:19 -07:00
David Kyle 6c849534da [ML][TEST] Clean up jobs after SE test
Original commit: elastic/x-pack-elasticsearch@c71aafeb06
2018-01-30 14:34:36 +00:00
Tim Vernum 427da8de8b Switch SAML QA test to use a standalone cluster (elastic/x-pack-elasticsearch#3743)
Introduce a healthy dose of reality into the SAML integration tests.
Switches the test class from *Tests to *IT, and updates the build to run it during integTest

Original commit: elastic/x-pack-elasticsearch@edd2538e5f
2018-01-30 23:03:04 +11:00
David Kyle 36ff4185a7 [ML] Move ML classes from core (elastic/x-pack-elasticsearch#3780)
* Move ML query builders

* Move autodetect params

* Remove unused MLClientActionPlugin

Original commit: elastic/x-pack-elasticsearch@23168c70dc
2018-01-30 11:55:30 +00:00
David Kyle edba6e329a [ML] Move ML action unit tests into same package as the classes (elastic/x-pack-elasticsearch#3772)
* Move ML action unit tests into same package as the classes

Original commit: elastic/x-pack-elasticsearch@9556d3c98a
2018-01-30 11:11:28 +00:00
David Kyle 10fae5e58f [ML][TEST] Scheduled events integration test (elastic/x-pack-elasticsearch#3555)
* Scheduled events integration test

Original commit: elastic/x-pack-elasticsearch@8181645fe5
2018-01-30 10:38:13 +00:00
Alexander Reelsen c65528b9f6 Watcher: Ensure state is cleaned properly in watcher life cycle service (elastic/x-pack-elasticsearch#3770)
The WatcherLifeCycleService is responsible for deciding if watches need
to be reloaded while running. In order to do this, the service stores
the currently local shard allocation ids in a List.

This data structure however was not properly updated all the time, when
it should have been - for example when a master node is not available.

This lead to unintended reloads, even though there was no change in the
allocated shards. This in turn lead to unwanted executions and unwanted
loading of triggered watches.

This commit should also fix one of the more nasty ongoing test failures,
where the test returns with an exception that only parts of watcher have
been started. The AbstractWatcherIntegrationTestCase now properly waits
until watcher is started before starting the test case itself.

Original commit: elastic/x-pack-elasticsearch@097f12a900
2018-01-30 10:28:48 +01:00
Tim Vernum da7560a079 [Security] Correct DN matches in role-mapping rules (elastic/x-pack-elasticsearch#3704)
Changes the behaviour of the role mapping API to perform a "DistinguishedNameMatch"
when the field is a DN. This is achieved by moving the responsibility for defining
the matching rules from the expression to the data (ExpressionModel)

Because the role mapping API is used within the SAML realm, which may or may not be
using DNs, this implementation assumes that the "dn" and "groups" should be
compared as DNs if they parse as a DN.
For SAML this behaviour will generally do the right thing, as members of the "groups"
field might be DNs (if the data is sourced from an LDAP directory) but often will not be.

Original commit: elastic/x-pack-elasticsearch@3a4dfbba79
2018-01-30 15:47:17 +11:00
jaymode 4271fd7cc3 Test: use System.lineseperator instead of `\n` in SamlSpMetadataBuilderTests
This commit fixes SamlSpMetadataBuilderTests#testBuildFullMetadata failures on windows due to
differing newline endings.

Original commit: elastic/x-pack-elasticsearch@3181c96e46
2018-01-29 13:47:42 -07:00
Simon Willnauer 32d4e7f9d0 [TEST] Use `ensureYellowAndNoInitializingShards` in monitoring test (elastic/x-pack-elasticsearch#3768)
Monitoring creates indices concurrently to the test execution. In that
case ensureYellow might not be enough and will cause test-failures when shards
are not active etc. This change uses a new method added in core to ensure shards
are not initializing anymore.

relates elastic/x-pack-elasticsearch#2672
Relates to elastic/elasticsearch#28416

Original commit: elastic/x-pack-elasticsearch@661e87f2ee
2018-01-29 20:46:35 +01:00
David Kyle b6bab14fd4 Revert "Mute ML rolling upgrade tests. Awaits fix elastic/x-pack-elasticsearch#1760"
This reverts commit elastic/x-pack-elasticsearch@deaf060818.

Original commit: elastic/x-pack-elasticsearch@2eece7d27e
2018-01-29 13:24:07 +00:00
Dimitris Athanasiou 664fcccb8b [ML][TEST] Add integ test for categorical rule (elastic/x-pack-elasticsearch#3739)
Original commit: elastic/x-pack-elasticsearch@975a10f810
2018-01-29 11:45:04 +00:00
Simon Willnauer e7b2102126 Remove extension executables
Relates to elastic/x-pack-elasticsearch#3734

Original commit: elastic/x-pack-elasticsearch@da75402bc9
2018-01-29 10:52:27 +01:00
Simon Willnauer 5fb3add5fd Remove XPackExtension break from 7.0 it's ported to 6.3
Original commit: elastic/x-pack-elasticsearch@5e8ebda481
2018-01-29 10:42:12 +01:00
Ryan Ernst 5a86450df7 Build: Replace references to x-pack-elasticsearch paths with helper methods (elastic/x-pack-elasticsearch#3748)
In order to more easily integrate xpack once it moves into the
elasticsearch repo, references to the existing x-pack-elasticsearch need
to be reduced. This commit introduces a few helper "methods" available
to any project within xpack (through gradle project extension
properties, as closures). All refeerences to project paths now use these
helper methods, except for those pertaining to bwc, which will be
handled in a followup.

Original commit: elastic/x-pack-elasticsearch@850668744c
2018-01-26 21:48:30 -08:00
Ryan Ernst e583b72c8d Build: Move and simplify ml cpp snapshot (elastic/x-pack-elasticsearch#3762)
Now that ML binaries are public, there is no longer a need to use the s3
client to access the bucket, since creds are not needed. This commit
also moves the cpp snapshot project under the ml module, since it is
specific to that and does not need to clutter the plugin dir.

Original commit: elastic/x-pack-elasticsearch@51e77da4ac
2018-01-26 20:14:07 -08:00
Dimitrios Athanasiou b2f0745c7b [ML][TEST] Use condition_type instead of type in rule bwc test
Original commit: elastic/x-pack-elasticsearch@f49fe28b4e
2018-01-26 18:46:11 +00:00
Ryan Ernst 0001cadbb5 Remove leftover license files (elastic/x-pack-elasticsearch#3746)
These were remaining from prior to the split into modules.

Original commit: elastic/x-pack-elasticsearch@97e05b5709
2018-01-26 10:22:18 -08:00
Ryan Ernst f43662686d Remove legacy files from license tools (elastic/x-pack-elasticsearch#3747)
This commit removes files leftover from when license tools were part of
the separate license plugin.

Original commit: elastic/x-pack-elasticsearch@c859b4c61a
2018-01-26 10:21:37 -08:00
David Roberts 9351b984ac [ML] Remove C++/Java handshake code (elastic/x-pack-elasticsearch#3757)
This is no longer needed following elastic/machine-learning-cpp#532

Original commit: elastic/x-pack-elasticsearch@76064b1539
2018-01-26 18:04:11 +00:00
Dimitrios Athanasiou 621f4f20b0 [ML][TEST] Add BWC tests for rules syntax change
Original commit: elastic/x-pack-elasticsearch@805d390417
2018-01-26 17:37:08 +00:00