Commit Graph

7994 Commits

Author SHA1 Message Date
Tim Vernum da7560a079 [Security] Correct DN matches in role-mapping rules (elastic/x-pack-elasticsearch#3704)
Changes the behaviour of the role mapping API to perform a "DistinguishedNameMatch"
when the field is a DN. This is achieved by moving the responsibility for defining
the matching rules from the expression to the data (ExpressionModel)

Because the role mapping API is used within the SAML realm, which may or may not be
using DNs, this implementation assumes that the "dn" and "groups" should be
compared as DNs if they parse as a DN.
For SAML this behaviour will generally do the right thing, as members of the "groups"
field might be DNs (if the data is sourced from an LDAP directory) but often will not be.

Original commit: elastic/x-pack-elasticsearch@3a4dfbba79
2018-01-30 15:47:17 +11:00
Lee Hinman 91cb994a83 [BUILD] use xpackModule for more things in SQL
Original commit: elastic/x-pack-elasticsearch@6139d949e5
2018-01-29 16:11:19 -07:00
Lee Hinman dc6b5ce453 [BUILD] have security's build.gradle use xpackModule for SQL dep
Original commit: elastic/x-pack-elasticsearch@88027b1d3b
2018-01-29 15:51:55 -07:00
Lee Hinman 7f3ba5c0ae Fix javadoc syntax in DataTypeConversion
Original commit: elastic/x-pack-elasticsearch@d7a2bbbe0f
2018-01-29 15:47:27 -07:00
Lee Hinman a789944b8d Merge remote-tracking branch 'origin/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@f96f80f0d0
2018-01-29 15:42:01 -07:00
Igor Motov 1627ec1378 SQL: Consolidate JDBC type handling (elastic/x-pack-elasticsearch#3761)
Consolidates handling of JDBC types conversion into a single file that should simplify maintaining consistency between type handling. Also separates the types that are handled as part of Elasticsearch output and types that are handled as user-supplied parameters.

relates elastic/x-pack-elasticsearch#3556

Original commit: elastic/x-pack-elasticsearch@d251fce66b
2018-01-29 16:35:42 -05:00
jaymode 4271fd7cc3 Test: use System.lineseperator instead of `\n` in SamlSpMetadataBuilderTests
This commit fixes SamlSpMetadataBuilderTests#testBuildFullMetadata failures on windows due to
differing newline endings.

Original commit: elastic/x-pack-elasticsearch@3181c96e46
2018-01-29 13:47:42 -07:00
Simon Willnauer 32d4e7f9d0 [TEST] Use `ensureYellowAndNoInitializingShards` in monitoring test (elastic/x-pack-elasticsearch#3768)
Monitoring creates indices concurrently to the test execution. In that
case ensureYellow might not be enough and will cause test-failures when shards
are not active etc. This change uses a new method added in core to ensure shards
are not initializing anymore.

relates elastic/x-pack-elasticsearch#2672
Relates to elastic/elasticsearch#28416

Original commit: elastic/x-pack-elasticsearch@661e87f2ee
2018-01-29 20:46:35 +01:00
Costin Leau 07658cc04f SQL: replace JDBC transport meta columns with SQL call (elastic/x-pack-elasticsearch#3740)
Replace meta column endpoint with SYS COLUMNS command

Original commit: elastic/x-pack-elasticsearch@819874bc5b
2018-01-29 19:18:24 +02:00
David Kyle b6bab14fd4 Revert "Mute ML rolling upgrade tests. Awaits fix elastic/x-pack-elasticsearch#1760"
This reverts commit elastic/x-pack-elasticsearch@deaf060818.

Original commit: elastic/x-pack-elasticsearch@2eece7d27e
2018-01-29 13:24:07 +00:00
Dimitris Athanasiou 664fcccb8b [ML][TEST] Add integ test for categorical rule (elastic/x-pack-elasticsearch#3739)
Original commit: elastic/x-pack-elasticsearch@975a10f810
2018-01-29 11:45:04 +00:00
Simon Willnauer e7b2102126 Remove extension executables
Relates to elastic/x-pack-elasticsearch#3734

Original commit: elastic/x-pack-elasticsearch@da75402bc9
2018-01-29 10:52:27 +01:00
Simon Willnauer 5fb3add5fd Remove XPackExtension break from 7.0 it's ported to 6.3
Original commit: elastic/x-pack-elasticsearch@5e8ebda481
2018-01-29 10:42:12 +01:00
Costin Leau e929d16d76 SQL: Replace ListTable transport with SQL command (elastic/x-pack-elasticsearch#3675)
Introduce system commands as alternative to meta HTTP endpoints
Pass in cluster name
Use 'BASE TABLE' instead of 'INDEX' when describing a table to stick
with the SQL terminology

Original commit: elastic/x-pack-elasticsearch@600312b8f7
2018-01-27 11:54:19 +02:00
Ryan Ernst 5a86450df7 Build: Replace references to x-pack-elasticsearch paths with helper methods (elastic/x-pack-elasticsearch#3748)
In order to more easily integrate xpack once it moves into the
elasticsearch repo, references to the existing x-pack-elasticsearch need
to be reduced. This commit introduces a few helper "methods" available
to any project within xpack (through gradle project extension
properties, as closures). All refeerences to project paths now use these
helper methods, except for those pertaining to bwc, which will be
handled in a followup.

Original commit: elastic/x-pack-elasticsearch@850668744c
2018-01-26 21:48:30 -08:00
Ryan Ernst e583b72c8d Build: Move and simplify ml cpp snapshot (elastic/x-pack-elasticsearch#3762)
Now that ML binaries are public, there is no longer a need to use the s3
client to access the bucket, since creds are not needed. This commit
also moves the cpp snapshot project under the ml module, since it is
specific to that and does not need to clutter the plugin dir.

Original commit: elastic/x-pack-elasticsearch@51e77da4ac
2018-01-26 20:14:07 -08:00
Dimitrios Athanasiou b2f0745c7b [ML][TEST] Use condition_type instead of type in rule bwc test
Original commit: elastic/x-pack-elasticsearch@f49fe28b4e
2018-01-26 18:46:11 +00:00
Ryan Ernst 0001cadbb5 Remove leftover license files (elastic/x-pack-elasticsearch#3746)
These were remaining from prior to the split into modules.

Original commit: elastic/x-pack-elasticsearch@97e05b5709
2018-01-26 10:22:18 -08:00
Ryan Ernst f43662686d Remove legacy files from license tools (elastic/x-pack-elasticsearch#3747)
This commit removes files leftover from when license tools were part of
the separate license plugin.

Original commit: elastic/x-pack-elasticsearch@c859b4c61a
2018-01-26 10:21:37 -08:00
David Roberts 9351b984ac [ML] Remove C++/Java handshake code (elastic/x-pack-elasticsearch#3757)
This is no longer needed following elastic/machine-learning-cpp#532

Original commit: elastic/x-pack-elasticsearch@76064b1539
2018-01-26 18:04:11 +00:00
Dimitrios Athanasiou 621f4f20b0 [ML][TEST] Add BWC tests for rules syntax change
Original commit: elastic/x-pack-elasticsearch@805d390417
2018-01-26 17:37:08 +00:00
David Kyle 50ea5f9cf8 [ML] Update Javadoc for expected behaviour in NameResolver (elastic/x-pack-elasticsearch#3733)
Update Javadoc

Original commit: elastic/x-pack-elasticsearch@e47f38b2ce
2018-01-26 16:03:31 +00:00
Dimitris Athanasiou 867b4fe9fc [ML] Ensure BWC between old and new rules syntax (elastic/x-pack-elasticsearch#3738)
relates elastic/x-pack-elasticsearch#3325

Original commit: elastic/x-pack-elasticsearch@74cccbccf0
2018-01-26 15:52:01 +00:00
Simon Willnauer 730e7075ab Remove XPackExtension in favor of SecurityExtensions (elastic/x-pack-elasticsearch#3734)
This change removes the XPackExtension mechanism in favor of
SecurityExtension that can be loaded via SPI and doesn't need
another (duplicate) plugin infrastructure

Original commit: elastic/x-pack-elasticsearch@f39e62a040
2018-01-26 16:14:11 +01:00
Lisa Cawley 7bec4af206 [DOCS] Clarified use of historical events (elastic/x-pack-elasticsearch#3737)
Original commit: elastic/x-pack-elasticsearch@371ca281e9
2018-01-26 06:56:36 -08:00
markharwood 620fe16ec4 [TEST] Remove unreliable scoring check on Graph API (elastic/x-pack-elasticsearch#918)
Original commit: elastic/x-pack-elasticsearch@3ba1340c70
2018-01-26 14:44:02 +00:00
Alexander Reelsen 49ecd23eb9 Tests: Ensure watcher is stopped before restarting
The EmailSecretsIntegrationTests did not properly wait for watcher to be
stopped before starting it again.

This test failed a few times per week across master and 6.x branches.

Original commit: elastic/x-pack-elasticsearch@cf20d58b0b
2018-01-26 11:17:08 +01:00
David Kyle 2356830782 [ML][TEST] Disable monitoring for ML QA tests (elastic/x-pack-elasticsearch#3751)
Original commit: elastic/x-pack-elasticsearch@4c3192e964
2018-01-26 10:10:52 +00:00
Ryan Ernst a8bce19aa9 Remove the gradle cheatsheet (elastic/x-pack-elasticsearch#3708)
The cheatsheet was removed from the elasticsearch repo in
https://github.com/elastic/elasticsearch/pull/28113. This commit removes
it from xpack for the same reasons.

Original commit: elastic/x-pack-elasticsearch@9bf50eeabd
2018-01-25 23:03:44 -08:00
Ryan Ernst 12d6bf2f09 Remove legacy files from xpack split (elastic/x-pack-elasticsearch#3707)
This commit removes leftover files from the split of xpack into project
specific repos.

Original commit: elastic/x-pack-elasticsearch@ba2eb749ec
2018-01-25 23:03:03 -08:00
Lisa Cawley 79e79dc467 [DOCS] Fixed broken link to monitoring info (elastic/x-pack-elasticsearch#3745)
Original commit: elastic/x-pack-elasticsearch@dacc2b82a2
2018-01-25 16:52:33 -08:00
Ryan Ernst 9caebebe65 Build: Remove vault/s3 auth for ml artifacts (elastic/x-pack-elasticsearch#3742)
The ML snapshot artifacts bucket is now public, so we no longer need to
grab generated s3 creds from vault. This makes the download task run
noticably faster.

Original commit: elastic/x-pack-elasticsearch@e680e55f3d
2018-01-25 16:17:41 -08:00
Lisa Cawley e447ea9c1f [DOCS] Move monitoring configuration info (elastic/x-pack-elasticsearch#3674)
Original commit: elastic/x-pack-elasticsearch@eb68a4d3ae
2018-01-25 13:49:30 -08:00
Jay Modi f07a850ba2 [SAML] add security permission to get the classloader (elastic/x-pack-elasticsearch#3720)
This adds the security permission to get the thread context classloader to the security
plugin after it was inadvertently removed in elastic/x-pack-elasticsearch#3651. This was not caught as there is a
dependency on monitoring for security and monitoring has the getClassLoader permission.

relates elastic/x-pack-elasticsearch#3719

Original commit: elastic/x-pack-elasticsearch@61ad950f5b
2018-01-25 13:01:48 -07:00
Lee Hinman dc40562be6 SQL: Remove cli.debug system parameter (elastic/x-pack-elasticsearch#3722)
This was requested for removal in
https://github.com/elastic/x-pack-elasticsearch/pull/3686#discussion_r163286953
and is a followup for that.

Original commit: elastic/x-pack-elasticsearch@6134693a34
2018-01-25 10:51:12 -07:00
Michael Basnight e59bcd8f8e Enable the licenseHeaders task for plugins (elastic/x-pack-elasticsearch#3673)
Original commit: elastic/x-pack-elasticsearch@c9949b8ca5
2018-01-25 11:34:02 -06:00
Lisa Cawley 705804ee0c [DOCS] Compacted setup scripts in build.gradle (elastic/x-pack-elasticsearch#3573)
Original commit: elastic/x-pack-elasticsearch@265950c186
2018-01-25 09:33:21 -08:00
Lisa Cawley b6c9e96304 [DOCS] Added job ID requirements (elastic/x-pack-elasticsearch#3727)
Original commit: elastic/x-pack-elasticsearch@71f0073708
2018-01-25 09:23:56 -08:00
Michael Basnight 39b4587d02 Remove requiresKeystore when not actually required (elastic/x-pack-elasticsearch#3697)
Original commit: elastic/x-pack-elasticsearch@6bdd4ae2dc
2018-01-25 07:33:30 -06:00
Jim Ferenczi 3316c964c8 Move watcher's Cron into x-pack core (elastic/x-pack-elasticsearch#3716)
This change moves watcher's Cron class in the schedule package of xpack-core so that it can be used by other projects (rollup).

Original commit: elastic/x-pack-elasticsearch@f0aa32ccc2
2018-01-25 11:57:44 +01:00
Ryan Ernst 3e24ea3ed2 Test: Add internalClusterTest to xpack modules (elastic/x-pack-elasticsearch#3730)
This commit reenables running ITs in xpack by adding an internalClusterTest to xpack modules that contain ESIntegTestCase tests. The new task allows us to run these independently of rest integ tests, which are disabled for xpack modules because installing the bundled plugins directly is not quite the same as installing via the meta plugin. Some tests (ML) are moved to their own qa module to accommodate the need for a real cluster. A couple tests (monitoring and upgrade) have been marked as AwaitsFix.

Commits that have been folded into this commit:
* Move ML IT tests to qa/ml-native-tests
* Add internalClusterTest task and disable rest integ tests for xpack
modules. Also tweak ML tests and get upgrade tests working
* Adding the keystore and security back to the ml native tests
* Fixing native integ test
* Fix last ML test, add awaits fix to monitoring and upgrade tests
* cleanup PR
* fix checkstyle

Original commit: elastic/x-pack-elasticsearch@3c0ed6fd3b
2018-01-25 10:38:34 +01:00
Tanguy Leroux 7c90f874d7 Update x-pack packaging tests to work with meta plugins (elastic/x-pack-elasticsearch#3683)
relates elastic/x-pack-elasticsearch#3652

Original commit: elastic/x-pack-elasticsearch@e177b742f9
2018-01-25 10:20:52 +01:00
jaymode b456885b39 Test: increase time to wait for tribe tests with security
There have been some failures in CI due to the tribe tests with security taking too long for the
cluster to form and the expected number of node to join. In the thread dumps of the failure, it
can be seen that a node is still initializing and is in a method that could take a decent amount
of time on slow machines (TokenService#computeSecretKey). This commit increases the wait time to
allow for the node to startup and join.

Original commit: elastic/x-pack-elasticsearch@2bf4c96d8f
2018-01-24 16:30:02 -07:00
Lee Hinman 4d213666d5 SQL: Move Exception classes to follow ES format (elastic/x-pack-elasticsearch#3721)
This commit moves the exception classes that SQL uses to follow the
Elasticsearch convention. In the places where varargs were used, the
`LoggerMessageFormat` (`{}`) standard is used instead.

In also removes on Exception - `ExecutionException` since it seemed to not ever
be beefed up, it can be re-added later if needed.

This removes the varargs version of `ClientException` to push the formatting
back on the caller, since `ClientException` cannot depend on Elasticsearch for
formatting the arguments.

There were also a couple of places where we incorrectly passed the Throwable
cause as a var-arg and were unintentionally swallowing it since `String.format`
discards unused arguments.

Relates to elastic/x-pack-elasticsearch#2880

Original commit: elastic/x-pack-elasticsearch@5f5d580e57
2018-01-24 13:02:01 -07:00
Jay Modi 3baf6f3a7a Remove production from the message about license installation without TLS (elastic/x-pack-elasticsearch#3666)
This change simply changes the wording of the message that is returned to the user when
installation of a license is attempted with security enabled and TLS disabled. The term
"production" has been removed as it means something different to users.

See elastic/x-pack-elasticsearch#2636

Original commit: elastic/x-pack-elasticsearch@9739c72d66
2018-01-24 12:44:25 -07:00
Igor Motov 36bd849cd3 SQL: Refactor the type handling (elastic/x-pack-elasticsearch#3694)
Separates ES types and field metadata.

Relates to elastic/x-pack-elasticsearch#3556

Original commit: elastic/x-pack-elasticsearch@c9601f3390
2018-01-24 14:43:23 -05:00
jaymode a547049303 Security: switch to IllegalArgumentException if request cannot be serialized
The commit switches the CreateTokenRequest and InvalidateTokenRequest to throw an
IllegalArgumentException when the version the request is being serialized to does not support the
request. This fixes test failures due to the use of the AssertingTransportInterceptor testing
serialization with versions prior to 6.2. The IAE is an indication to the
AssertingTranpsortInterceptor that the request does not support the version.

Original commit: elastic/x-pack-elasticsearch@c73abf1bc0
2018-01-24 12:19:09 -07:00
David Roberts ae1a90eba6 [ML] Move JobProvider and the Batched*Iterators out of core (elastic/x-pack-elasticsearch#3712)
By moving these into the ML module:

1. The classes are in the same module as their unit tests
2. We can extend the JobProvider in the future with functionality
   that is not in core

Original commit: elastic/x-pack-elasticsearch@610a89a3d9
2018-01-24 18:56:30 +00:00
Jay Modi 0baa45d9b3 [SAML] Find all tokens for a realm, not just the first 10 (elastic/x-pack-elasticsearch#3689)
This commit changes the token service to use a scroll based approach when finding all tokens by
the realm. Without this, we may only find a few tokens and leave some active that need to be
invalidated.

relates elastic/x-pack-elasticsearch#3688

Original commit: elastic/x-pack-elasticsearch@20e97b6aae
2018-01-24 11:07:51 -07:00
jaymode 2c46002c00 Test: get the rest-api-spec from the correct location
Original commit: elastic/x-pack-elasticsearch@a3f3c45213
2018-01-24 10:39:43 -07:00