Commit Graph

7690 Commits

Author SHA1 Message Date
Alexander Reelsen d5281e89b7 Watcher: Register HTML sanitizer settings (elastic/x-pack-elasticsearch#3904)
The HTML sanitizer settings were not registered correctly, when the
grouping functionality was removed in elastic/x-pack-elasticsearch#3685

This registers the settings properly in watcher.

Relates elastic/support-dev-help-elasticsearch#413

Original commit: elastic/x-pack-elasticsearch@787bb3d9d3
2018-02-12 15:34:04 +01:00
Alexander Reelsen 54cb890eb7 Watcher: Allow to defer startup for tests (elastic/x-pack-elasticsearch#3903)
This commit introduces a new watcher setting to defer starting watcher
until it has been called with the API for the first time. This is
primarily useful in testing environments, as this ensures that watcher
does not try to reload itself because of starting first and then
creating watcher indices.

In addition the undocumented and unused option
xpack.watcher.start_immediately has been removed.

Relates elastic/x-pack-elasticsearch#3854

Original commit: elastic/x-pack-elasticsearch@2b55aec4ad
2018-02-12 14:24:05 +01:00
Alexander Reelsen 48f6a752cb Watcher: Fix proxy scheme to default to HTTP (elastic/x-pack-elasticsearch#3844)
This fixes a regression introduced in Elasticsearch 6.0, when switching
from HttpURLConnection to the Apache HTTP Client.

In the old implementation there was no way to specify if you wanted to use HTTP
or HTTPS for your proxy, only HTTP. If people needed to use HTTPs, they
could just use the CONNECT feature of the proxy.

The new implementation used the scheme of the request that was about to
be sent out as the proxy scheme to be used. So if the request was HTTPS
but the proxy server was HTTP this created a problem.

This commit changes the default scheme to be just HTTP, so that then the
standard CONNECT procecure is taken care off.

Without a real proxy server this is super hard to test. I have verified
this with the following test against a tinyproxy running on port 8888,
but I do not have a great idea how to test this in a unit testable way using a real proxy.

Original commit: elastic/x-pack-elasticsearch@f68e72d8f1
2018-02-12 13:12:38 +01:00
Tanguy Leroux 8e501b6a7e [Tests] Add more time to start with X-Pack
This commit adds a bit more time to the packaging tests to start when
x-pack is installed.

Original commit: elastic/x-pack-elasticsearch@228a3d7c33
2018-02-12 11:35:46 +01:00
Alexander Reelsen f4fa274aa8 Tests: Do not set number of replicas for watcher index (elastic/x-pack-elasticsearch#3882)
The watcher index uses auto expanding of replicas nowadays, so there is
no need to set the number of replicas.

Also ensuring that all watcher indices are green on startup has been moved
to one call, in order to prevent indices being moved around after adding
the triggered watches or watch history index.

Original commit: elastic/x-pack-elasticsearch@dbe04adf64
2018-02-12 09:35:03 +01:00
Chris Earle 63c2d3e6db [Monitoring] Change MonitoringIT into ESIntegTestCase (elastic/x-pack-elasticsearch#3899)
Change MonitoringIT into ESIntegTestCase

Original commit: elastic/x-pack-elasticsearch@d06f6cc0b3
2018-02-10 12:37:17 -05:00
Michael Basnight f6b0de2536 Pull snapshots prior to 6.3 from elastic maven (elastic/x-pack-elasticsearch#3900)
Snapshot versions before 6.3 do not exist in the public, so they need to
be pulled from snapshots.elastic.co/maven.

Original commit: elastic/x-pack-elasticsearch@3cc94add91
2018-02-10 09:52:36 -06:00
Shaunak Kashyap 7ee5ef333c Add fields to `.logstash`'s mapping in template (elastic/x-pack-elasticsearch#3872)
* Add fields to `.logstash`'s mapping in template

This "makes room" in the index for pipeline settings and node groups. Due to this change, users will be able to specify settings and node groups for a pipeline via the Centralized Config Management UI in Kibana. Logstash will only retrieve pipelines associated with the node group specified via the `xpack.management.group.id` setting in `logstash.yml`. For the retrieved pipelines, Logstash will apply any (optionally) specified pipeline settings before (re)loading the pipelines.

* Making field name more explicit + adding multi field for better search

Original commit: elastic/x-pack-elasticsearch@2df101f0b1
2018-02-09 15:18:37 -08:00
Nik Everett 45ac81127a SQL: Drop requirement to have keystore (elastic/x-pack-elasticsearch#3896)
SQL doesn't need the keystore. We required it by mistake.

Original commit: elastic/x-pack-elasticsearch@6840637feb
2018-02-09 17:17:46 -05:00
Michael Basnight 24c6a21d8f BWC changes for x-pack (elastic/x-pack-elasticsearch#3841)
relates elastic/elasticsearch#28505

Original commit: elastic/x-pack-elasticsearch@eda97ded76
2018-02-09 15:03:08 -06:00
Chris Earle 9c3257e158 [Monitoring] Register affixed Header Setting (elastic/x-pack-elasticsearch#3893)
This adds the `HEADERS_SETTING` as setting reported by the Monitoring plugin, which was changed in 6.2.

It also adds an IT to ensure that it is not missed in the future (existing integration tests were passing in the Settings and not being validated by the cluster!).

Original commit: elastic/x-pack-elasticsearch@d6f0b4b503
2018-02-09 15:53:27 -05:00
Ryan Ernst 4e578fe7cc Build: Replace provided configuration with compileOnly (elastic/x-pack-elasticsearch#3868)
This is the xpack side of elastic/elasticsearch#28564

Original commit: elastic/x-pack-elasticsearch@60033824a4
2018-02-09 11:30:43 -08:00
Costin Leau 3c6f9d4d29 SQL: fix name of Arithmetic functions (elastic/x-pack-elasticsearch#3884)
* SQL: fix name of Arithmetic functions

Remove id from name of Arithmetic function and improve name of Neg

Original commit: elastic/x-pack-elasticsearch@4f3e8d6a2d
2018-02-09 19:48:22 +02:00
Ioannis Kakavas 7504e33da7 Change value of METADATA_REFRESH for the tests (elastic/x-pack-elasticsearch#3879)
Ensure that the next triggered metadata refresh is always in the
future. relates elastic/x-pack-elasticsearch#3815

Original commit: elastic/x-pack-elasticsearch@edd9ba911d
2018-02-09 18:22:26 +01:00
Lisa Cawley 4e0c1d1b60 [DOCS] Enabled more ML code snippet testing (elastic/x-pack-elasticsearch#3764)
Original commit: elastic/x-pack-elasticsearch@518dce3ddd
2018-02-09 09:16:24 -08:00
Martijn van Groningen d2dc630e84 Changes required for elastic/elasticsearch#28588
Original commit: elastic/x-pack-elasticsearch@11d11c2d39
2018-02-09 17:14:30 +01:00
Lee Hinman dc833f2448 Remove all instances of the deprecated `ParseField.match` method (elastic/x-pack-elasticsearch#3874)
* Remove all instances of the deprecated `ParseField.match` method

This removes all the server references to the deprecated `ParseField.match`
method in favor of the method that passes in the deprecation logger.

Relates to https://github.com/elastic/elasticsearch/issues/28504

* Fix line-length

Original commit: elastic/x-pack-elasticsearch@e7cb2611f7
2018-02-09 09:01:18 -07:00
Alexander Reelsen 98e9365aa1 Tests: Increase logging for GetWatchTests
In order to get more insights into a specific test failure, that we
suppose to be in core, increase search logging to debug level.

Relates elastic/x-pack-elasticsearch#3854

Original commit: elastic/x-pack-elasticsearch@5c37535e03
2018-02-09 16:37:33 +01:00
Albert Zaharovits fb8adb4a1c [Security][Tests] Azeri(Turkish) locale tripps opensaml dependency
Original commit: elastic/x-pack-elasticsearch@9c4ecabe48
2018-02-09 13:57:09 +02:00
Alexander Reelsen aa12763eba Docs: Remove invalid configuration options
Original commit: elastic/x-pack-elasticsearch@c1ef6dce5f
2018-02-09 09:35:56 +01:00
Ryan Ernst 8d6f402ac9 Fix silly monitoring test checking exact output
Original commit: elastic/x-pack-elasticsearch@58c23551f6
2018-02-08 14:39:03 -08:00
Ryan Ernst a2c9c97650 Fix usage of PluginInfo
See https://github.com/elastic/elasticsearch/pull/28556

Original commit: elastic/x-pack-elasticsearch@1e14b14526
2018-02-08 09:51:20 -08:00
David Kyle d6ca53cd26 [ML] Fix exception categorising an empty string (elastic/x-pack-elasticsearch#3870)
Original commit: elastic/x-pack-elasticsearch@1840a74415
2018-02-08 16:18:48 +00:00
David Kyle 8e73085047 [ML] Enable adding multiple jobs to a calendar (elastic/x-pack-elasticsearch#3786)
Original commit: elastic/x-pack-elasticsearch@56a70a4580
2018-02-08 11:44:16 +00:00
Albert Zaharovits a57999e1e1 [Security][Tests] Print LDAP reconnect stacktrace on fail
Original commit: elastic/x-pack-elasticsearch@18f235197c
2018-02-08 13:41:02 +02:00
markharwood bce8d6f9e6 Test: Remove off-by-one test logic that fails every one in several thousand tests. relates elastic/x-pack-elasticsearch#270 and elastic/x-pack-elasticsearch#869 (which is also now re-enabled).
relates elastic/x-pack-elasticsearch#270

Original commit: elastic/x-pack-elasticsearch@2e52efd601
2018-02-08 10:48:59 +00:00
David Roberts f4b32bef8c [ML] Reject headers supplied directly in the put datafeed body (elastic/x-pack-elasticsearch#3853)
Since elastic/x-pack-elasticsearch#3254 security headers have been stored in datafeed cluster state
to allow the datafeed to run searches using the credentials of the user
who created/updated it.  As a result the parser was changed to read the
"headers" field so that cluster state could be reloaded.  However, this
meant that datafeed configs could be submitted with a "headers" field.
No security loophole arose from this, as subsequent code overwrites the
contents of any supplied headers.  But it could be confusing that an
erroneously supplied field did not cause a parse failure as it usually
would.

This change makes the config parser for datafeeds reject a "headers"
field.  Now only the metadata parser used for reloading cluster state
will read a "headers" field.

Original commit: elastic/x-pack-elasticsearch@afa503275f
2018-02-08 10:43:33 +00:00
Alexander Reelsen 064a0819d9 Cleanup: Remove HaltedClock (elastic/x-pack-elasticsearch#3731)
java.time features it's own halted clock, called a fixed clock, we can
use that one.

On top of that the watcher xcontent parser does not need a clock at all,
just a timestamp when parsing happened.

Original commit: elastic/x-pack-elasticsearch@2061aeffe1
2018-02-08 09:24:08 +01:00
Tim Vernum 3102b94946 [TEST] Don't fail IndexAuditTrailTests if the test runs very fast
Relates: elastic/x-pack-elasticsearch#3472

Original commit: elastic/x-pack-elasticsearch@00fde8973f
2018-02-08 16:44:58 +11:00
Ryan Ernst 7d71c3f1cc Build: Remove refs to x-pack-elasticsearch (elastic/x-pack-elasticsearch#3863)
These were removed before, but sql added a bunch of new hard references
to the project name.

Original commit: elastic/x-pack-elasticsearch@2eb156fd67
2018-02-07 20:50:12 -08:00
Ryan Ernst 8ee3de122b Build: Remove x-pack-api jar (elastic/x-pack-elasticsearch#3864)
The api jar was added for xpack extensions. However, extensions have
been removed in favor of using SPI, and the individual xpack jars like
core and security are published to enable this. This commit removes the
api jar, and switches the transport client to use the core jar (which
the api jar was just a rename of).

Original commit: elastic/x-pack-elasticsearch@58e069e66c
2018-02-07 19:21:43 -08:00
Tim Vernum 736cc05d72 [DOCS] Fix broken format in SAML guide
Original commit: elastic/x-pack-elasticsearch@6d268e91f0
2018-02-08 12:37:08 +11:00
Ryan Ernst f360d2d8d7 Build: Cleanup gradle files (elastic/x-pack-elasticsearch#3867)
There were a number of leftover unnecessary elements in the module
build.gradle files that were holdovers from copying the original plugin
build.gradle. This commit removes these elements.

Original commit: elastic/x-pack-elasticsearch@08babbd520
2018-02-07 16:58:09 -08:00
Tim Vernum 256ef79cba [TEST] QA resources should not need vagrant provision (elastic/x-pack-elasticsearch#3851)
For the idp-fixture (OpenLDAP + SAML), we have been generating the CA as part of the provisioning steps for the VM and then adding it to the test resources for the gradle project.

This meant that test-resources were dependent on vagrant provision, and as a consequence vagrant would download and provision the box during precommit. A bad thing (TM)

This change introduces a pre-generated CA, which is supplied to the VM instead so the tests only depend on fixed resources.
(The SAML integration test still uses the generated IdP Metadata file, but it copies it as part integ-test cluster setup, and doesn't treat it as a gradle "test resource")

Original commit: elastic/x-pack-elasticsearch@a352bf2a1f
2018-02-08 11:42:25 +11:00
Nik Everett f15189c9e5 SQL: Re-add removed CI script behavior
When I dropped the CI script changes for SQL I forgot to restore the old
behavior. This fixes that.

Original commit: elastic/x-pack-elasticsearch@0e18b41464
2018-02-07 15:03:42 -05:00
Nik Everett 8c9f7886fe SQL: Drop changes to CI script
They were hacks for SQL and will break others.

Original commit: elastic/x-pack-elasticsearch@c3a997f974
2018-02-07 13:05:58 -05:00
Nik Everett f0f8fb6e5f SQL: Fix doc pointer to SQL's tests
The docs include portions of the SQL tests and for that to work they
need to point to position of the tests. They use a relative directory
but relative to *what*? That turns out to be a fairly complex thing to
answer, luckilly, `index.x.asciidoc` defines `xes-repo-dir` which points
to the root of the xpack docs. We can use that to find the sql tests
without having to answer the "relative to what?" question in two places.

Original commit: elastic/x-pack-elasticsearch@ebea586fdf
2018-02-07 12:57:11 -05:00
Lisa Cawley 7a070568d3 [DOCS] Expanded conceptual information about datafeeds (elastic/x-pack-elasticsearch#3849)
Original commit: elastic/x-pack-elasticsearch@09a4229ed9
2018-02-07 09:29:22 -08:00
Jason Tedor 6a6ae24541 Remove transport client dependency on security
This commit removes the compile-time dependency of the transport client
on security. This appears to have been inadvertent.

Relates elastic/x-pack-elasticsearch#3860

Original commit: elastic/x-pack-elasticsearch@c53d5dde1e
2018-02-07 11:48:40 -05:00
Martijn van Groningen eb2fce9d54 Changes required for elastic/elasticsearch#28531
Original commit: elastic/x-pack-elasticsearch@e77398e93f
2018-02-07 17:45:33 +01:00
Nik Everett b95745956e Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@5f6bbb588a
2018-02-07 09:38:13 -05:00
Tim Vernum 7bb2cba14f [Security] Reset IndexAuditTrail to INITIALISED before start (elastic/x-pack-elasticsearch#3807)
Calling start() when already in the STARTING state doesn't do anything, so the component
gets stuck in STARTING state forever.

Also: wait on the required index name not just the cluster.

Also: added more logging to help diagnose such issues (either in RemoteIndexAuditTrailStartingTests or production)

Original commit: elastic/x-pack-elasticsearch@fb81214fe7
2018-02-07 12:30:23 +11:00
Nik Everett 10efd82694 SQL: Shift sql-cli into its proper home
The old home made sense before x-pack was split. The new home lines up
with where security keeps its scripts.

The jar file wasn't being included any more so this re-adds that.

Original commit: elastic/x-pack-elasticsearch@d3ec941397
2018-02-06 16:17:18 -05:00
jaymode 092fcd3675 Test: do not check cluster stats in audit qa project
This commit wraps the TestCluster in one that does not check stats to prevent tripping an
assertion for the accounting breaker as the index audit trail is running in the background and
will affect the value of the breakers.

See elastic/x-pack-elasticsearch#157

Original commit: elastic/x-pack-elasticsearch@381ebbd413
2018-02-06 13:35:09 -07:00
jaymode 4731a1b56b Test: disable monitoring for security spi qa project
This commit disables monitoring for the qa project that tests custom realms and role providers
as monitoring can cause failures due to an accounting breaker not being reset.

See elastic/x-pack-elasticsearch#157

Original commit: elastic/x-pack-elasticsearch@f882507e85
2018-02-06 13:09:09 -07:00
Nik Everett a2f1ed3f62 SQL: Fix javadoc in JdbcConfiguration
It wasn't building....

Original commit: elastic/x-pack-elasticsearch@059c6397a2
2018-02-06 14:41:06 -05:00
Nik Everett 5b46899cc3 Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@9a49ecbbd5
2018-02-06 14:40:55 -05:00
Jay Modi 18c04c714d [Security] handle null values when attempting to parse as DN when mapping roles (elastic/x-pack-elasticsearch#3791)
This commit adds special handling for null values when building a DistinguishedNamePredicate for
role mapping. Previously this would have resulted in an exception from the unboundid ldapsdk as
the DN is invalid.

relates elastic/x-pack-elasticsearch#3787

Original commit: elastic/x-pack-elasticsearch@9386dae03e
2018-02-06 10:19:49 -07:00
Nik Richers 20c1791dd8 Fix broken link for 6.2
Direct commit to speed up fix.

Original commit: elastic/x-pack-elasticsearch@f9187b81b8
2018-02-06 07:21:19 -08:00
Lee Hinman b2b95cb25d [TEST] Fix compilation for XContent parser
Original commit: elastic/x-pack-elasticsearch@bf03cc34fe
2018-02-05 16:56:36 -07:00