Commit Graph

7307 Commits

Author SHA1 Message Date
Nik Everett c78328853f Remove SQL ci script changes
These were useful when SQL lived on its own branch but aren't now that
we've merged SQL to master.

Original commit: elastic/x-pack-elasticsearch@afb4347465
2017-12-12 14:49:46 -05:00
Nik Everett aef5ea6527 Merge branch 'feature/sql'
It is time.

Original commit: elastic/x-pack-elasticsearch@2b3f7697a5
2017-12-12 14:49:27 -05:00
Jay Modi d48ad6dbd6 Enhance security for alias and resize operations (elastic/x-pack-elasticsearch#3302)
This commit adds additional checks around resize operations and alias creation operations to
add an extra layer of security around these APIs.

Original commit: elastic/x-pack-elasticsearch@b79f16673c
2017-12-12 12:01:27 -07:00
Nik Everett eaa75cfdbb Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b5fc2b484d
2017-12-12 11:29:23 -05:00
Nik Everett 2ca729afc2 Fix packaging tests after breaking up x-pack
The tests were failing after breaking up x-pack. This makes them pass
locally.

Original commit: elastic/x-pack-elasticsearch@af2a7866eb
2017-12-12 11:05:51 -05:00
Dimitris Athanasiou 92c40061c5 [ML] Skip index health checks for remote indices on datafeed node sel (elastic/x-pack-elasticsearch#3301)
Upon selecting a node to run a datafeed we normally check that the
data indices exists and their primaries are active. However, these
checks cannot be applied for CCS to a remote cluster. This commit
skips these checks for remote indices.

This removes the last obstacle for running CCS datafeeds.

Relates elastic/x-pack-elasticsearch#1071

Original commit: elastic/x-pack-elasticsearch@092f44feee
2017-12-12 15:12:33 +00:00
Tanguy Leroux 711254fd24 [Monitoring] Use the same Cluster State for all Collectors (elastic/x-pack-elasticsearch#3216)
This commit changes the Collectors so that they all use the 
same instance of ClusterState.

relates elastic/x-pack-elasticsearch#3156

Original commit: elastic/x-pack-elasticsearch@4f537b026c
2017-12-12 12:29:40 +01:00
David Kyle 0d46e9035c [ML] Get Filters should use executeAsyncWithOrigin (elastic/x-pack-elasticsearch#3295)
* Get Filters should use executeAsyncWithOrigin

Original commit: elastic/x-pack-elasticsearch@786c7bfd06
2017-12-12 11:15:54 +00:00
David Kyle 6113b86bdb [ML] Special events calendar CRUD endpoints (elastic/x-pack-elasticsearch#3267)
* Calendar CRUD endpoints

* Get calendars requires monitor permission

* Address review comments

* Add page params to get calendars

Original commit: elastic/x-pack-elasticsearch@badd1e6add
2017-12-12 09:21:44 +00:00
Nik Everett 6d56616d98 SQL: Ignore environment in CliFixture
If the currenct environment contains `JAVA_TOOLS` then the cli tests
will fail because they do not expect the notification about `JAVA_TOOLS`
that java prints to stdout. The clears the environment before starting
the CLI subprocess in an effort to remove the unexpected output.

Original commit: elastic/x-pack-elasticsearch@5729c1c09b
2017-12-11 16:42:52 -05:00
Nik Everett c27332be4a SQL: Fix docs build and cleanup docs index for merge
relates elastic/x-pack-elasticsearch#3084

Original commit: elastic/x-pack-elasticsearch@a64de41fbc
2017-12-11 16:39:18 -05:00
Nik Everett 2d9ce6132b SQL: Prepare docs for merge
Cleans up the index.asciidoc file so it merges cleanly with master and
builds SQL's docs.

Original commit: elastic/x-pack-elasticsearch@3c63c0bda0
2017-12-11 16:15:31 -05:00
Nik Everett 15f84840f2 SQL: Revert some security changes so NOOP merge
SQL used to have some changes to security. We've since reverted them but
we have some leftover stuff like import reordering and spacing changes.
We may as well remove them so merging SQL to master is smaller.

Original commit: elastic/x-pack-elasticsearch@c632256ddd
2017-12-11 16:06:48 -05:00
Nik Everett 236f64a70e SQL: Fix SSL for JDBC and CLI for real this time (elastic/x-pack-elasticsearch#3277)
Previously I'd added tests for JDBC and CLI that I *thought* used SSL but they didn't! I wasn't careful...

Testing changes:
* Actually enable SSL/HTTPS in the `qa:sql:security:ssl` subproject.
* Rework how `RemoteCli` handles security. This allows us to configure SSL, the keystore, and the username and password in a much less error prone way.
* Fix up JDBC tests to properly use SSL.
* Allow the `CliFixture` to specify the keystore location.
* Switch `CliFixture` and `RemoteCli` from sending the password in the connection string to filling out the prompt for it.
* Have `CliFixture` also send the keystore password when a keystore is configured.

This makes the following production code changes:
* Allow the CLI to configure the keystore location with the `-k`/`-keystore_location` parameters.
* If the keystore location is configured then the CLI will prompt for the password.
* Allow the configuration of urls starting with `https`.
* Improve the exception thrown when the URL doesn't parse by adding a suppressed exception with the original parse error, before we tried to add `http://` to the front of it.

Original commit: elastic/x-pack-elasticsearch@97fac4a3b4
2017-12-11 15:45:34 -05:00
David Roberts 249d06b256 [ML] Fix permissions for field caps in scroll data extractor
Follow up to elastic/x-pack-elasticsearch#3254

Original commit: elastic/x-pack-elasticsearch@d4df9446c0
2017-12-11 17:45:32 +00:00
Igor Motov 4bebc307c3 SQL: Add ability to close cursors (elastic/x-pack-elasticsearch#3249)
This commits adds a new end point for closing in-flight cursors, it also ensures that all cursors are properly closed by adding after test checks that ensures that we don't leave any search context open.

relates elastic/x-pack-elasticsearch#2878

Original commit: elastic/x-pack-elasticsearch@1052ea28dc
2017-12-11 11:36:02 -05:00
David Kyle 04c07688a2 [ML] Align special events with buckets (elastic/x-pack-elasticsearch#3258)
Original commit: elastic/x-pack-elasticsearch@71f9d0fb13
2017-12-11 15:42:06 +00:00
Igor Motov fab3712e3d SQL: Check connection on CLI startup (elastic/x-pack-elasticsearch#3278)
* SQL: Check connection on CLI startup

Adds a connection check at the CLI startup. If connection cannot be established or elasticsearch has incompatible version, the CLI doesn't start.

relates elastic/x-pack-elasticsearch#2984

Original commit: elastic/x-pack-elasticsearch@c9a58d2cd6
2017-12-11 10:00:00 -05:00
David Roberts 5fd68959a0 [ML] Make datafeeds run-as the user who created/updated them (elastic/x-pack-elasticsearch#3254)
This is the ML equivalent of what was done for Watcher in elastic/x-pack-elasticsearch#2808.

For security reasons, ML datafeeds should not run as the _xpack
user.  Instead, they record the security headers from the request
to create/update them, and reuse these when performing the search
to retrieve data for analysis.

Relates elastic/x-pack-elasticsearch#1071

Original commit: elastic/x-pack-elasticsearch@29f85de404
2017-12-11 13:01:16 +00:00
Alexander Reelsen 6bae4681e2 Tests: Replace http input integration test with REST test (elastic/x-pack-elasticsearch#3215)
Remove HttpInputIntegrationTests, which only tested existing
functionality, the remaining part was moved over into a REST tests.

relates elastic/x-pack-elasticsearch#3210

Original commit: elastic/x-pack-elasticsearch@474e5337b6
2017-12-11 13:08:58 +01:00
Jason Tedor e25d7db01a Quote version value in Docker Compose file
The example Docker Compose file does not quote the version value however
the Docker Compose documentation specifies this value should be quoted
to distinguish it from being treated as a number.

Relate elastic/elasticsearch#27745

Original commit: elastic/x-pack-elasticsearch@42ad68c3ac
2017-12-10 15:00:44 -05:00
Igor Motov 018d4d7722 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@a113c54b94
2017-12-09 12:21:07 -05:00
Tim Brooks 9dbb887a58 Update x-pack for change to channel tracking (elastic/x-pack-elasticsearch#3261)
This commit updates x-pack to be compatible with
elastic/elasticsearch#27711. That commit removed the need for channels
to be internally tracked inside transport implementations. This commit
removes a test mocking class that is not necessary after that change.

Original commit: elastic/x-pack-elasticsearch@75d99ba1d1
2017-12-08 16:57:08 -07:00
Lee Hinman 91d1812cb8 Rename x-pack-common -> x-pack-core (elastic/x-pack-elasticsearch#3272)
The folder has to be named 'x-pack-core' (not 'core'), otherwise Gradle flips
out and has circular dependencies.

Relates to elastic/x-pack-elasticsearch#3233

Original commit: elastic/x-pack-elasticsearch@f756e5d356
2017-12-08 14:42:50 -07:00
Igor Motov 5d0b8ff796 SQL: Rename Strings.splitStringByCommaToSet to tokenizeByCommaToSet
Relates elastic/x-pack-elasticsearch#3265

Original commit: elastic/x-pack-elasticsearch@8b06b0688b
2017-12-08 13:52:08 -05:00
Igor Motov d3cd462700 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@496f9cc46b
2017-12-08 13:22:20 -05:00
Tim Brooks a1d72fc004 Modify security transport to take PageCacheRecycler (elastic/x-pack-elasticsearch#3251)
This is related to elastic/elasticsearhc#27696. That commit introduced
signature changes to `NetworkPlugin.getTransports`. This commit updates
x-pack to be compatible with that change.

Original commit: elastic/x-pack-elasticsearch@2e331ef6cd
2017-12-08 10:39:41 -07:00
Igor Motov 4330195cdf Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@f4ef2d9cdf
2017-12-08 12:35:59 -05:00
Jason Tedor 27a3a80c07 Adapt to core Strings method rename
This commit adapts a single usage of a method that was renamed in core.

Relates elastic/x-pack-elasticsearch#3265

Original commit: elastic/x-pack-elasticsearch@107c495d63
2017-12-08 12:17:29 -05:00
Igor Motov b72a9b5674 SQL: switch from _sql/ endpoint to _xpack/sql endpoint (elastic/x-pack-elasticsearch#3270)
As a part of xpack, SQL should use _xpack/sql endpoint instead of _sql endpoint.

relates elastic/x-pack-elasticsearch#3114

Original commit: elastic/x-pack-elasticsearch@f561b57f16
2017-12-08 12:17:26 -05:00
Lisa Cawley 8a0e7f58b8 [DOCS] Split ML overview topics (elastic/x-pack-elasticsearch#3262)
Original commit: elastic/x-pack-elasticsearch@793e25ff62
2017-12-08 09:07:14 -08:00
Lee Hinman 24d91298db Create skeleton for plugin split (elastic/x-pack-elasticsearch#3233)
This creates a basic skeleton for the plugin split by adding folders and example
`build.gradle` files. It also includes a non-implemented `migrate-plugins.sh`
script that we can fill in at a later time.

Relates to elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@2ab035d6b6
2017-12-08 09:41:32 -07:00
Dimitris Athanasiou a9535c0b5a [ML][DOCS] Correct get-overall-buckets API example (elastic/x-pack-elasticsearch#3269)
Also fixes the score filters explanation for the results APIs.

Original commit: elastic/x-pack-elasticsearch@18cb31ab56
2017-12-08 16:03:51 +00:00
Ioannis Kakavas aff5cc840f Make generated passwords shell safe (elastic/x-pack-elasticsearch#3253)
Generate passwords from  [A-Za-z0-9] so that they are safe to be
used in shell scripts.
Entropy deterioration is not significant (124.9 -> 119), generated
passwords still meet guidelines and best practices regarding length
and complexity.

Resolves elastic/x-pack-elasticsearch#3087

Original commit: elastic/x-pack-elasticsearch@078639e7c2
2017-12-08 18:03:13 +02:00
Dimitrios Athanasiou 4082d6fa48 [Watcher] Remove line that accidentally sneaked in
Original commit: elastic/x-pack-elasticsearch@e63b1942ca
2017-12-08 13:47:53 +00:00
Dimitris Athanasiou 434dc94eb2 [ML] Avoid all overhead when renormalization window is zero (elastic/x-pack-elasticsearch#3255)
relates elastic/x-pack-elasticsearch#3244

Original commit: elastic/x-pack-elasticsearch@fe41c23ad7
2017-12-08 12:22:52 +00:00
Alexander Reelsen 3f17c28f9b Docs: Remove note about running watches with elevated privileges
The documentation still mentioned that watches are running with elevated
privileges, which is not true anymore since elastic/x-pack-elasticsearch#2808.

Original commit: elastic/x-pack-elasticsearch@6b510d999a
2017-12-08 11:01:28 +01:00
Yannick Welsch 9ddbef0641 Fix long path issue of bwc checkouts on Windows (elastic/x-pack-elasticsearch#3259)
Hopefully fixes the Windows CI failures that break on cloning the repository into a target directory with a lengthy path name.

Original commit: elastic/x-pack-elasticsearch@fe18e95d3f
2017-12-07 19:04:40 +01:00
Jason Tedor 0c6ec82317 Grant Netty necessary permissions
When using the security networking implementations, the Netty jars that
are in play are those that are loaded in the X-Pack classloader. This
means that permissions granted to the Netty jars loaded in the
transport-netty4 module classloader do nothing. Instead, we have to
grant the same permissions to the Netty jars in the X-Pack
classloader. This commit does this.

Relates elastic/x-pack-elasticsearch#3247

Original commit: elastic/x-pack-elasticsearch@91780597b9
2017-12-07 12:57:48 -05:00
Jay Modi c6799de2a4 Do not enforce TLS if discovery type is single-node (elastic/x-pack-elasticsearch#3245)
This commit adds a check for the discovery type so that the TLS join validator does not fail join
requests when the discovery type is single-node.

relates elastic/x-pack-elasticsearch#2828

Original commit: elastic/x-pack-elasticsearch@fdfdb76b0b
2017-12-07 09:50:25 -07:00
Lisa Cawley c7b100a8c3 [DOCS] Added X-Pack bootstrap checks (elastic/x-pack-elasticsearch#2747)
* [DOCS] Added X-Pack bootstrap checks

* [DOCS] Added more bootstrap checks

* [DOCS] Added link to TLS configuration

* [DOCS] Added Encrypt Sensitive Data bootstrap check

* [DOCS] Added link to watcher encryption task

* [DOCS] Added role mapping bootstrap check

* [DOCS] Added PKI realm boostrap check

* [DOCS] Added token SSL bootstrap check

* [DOCS] Added role mapping bootstrap check

* [DOCS] Added bootstrap check file names

Original commit: elastic/x-pack-elasticsearch@4f3eb72d94
2017-12-07 08:42:02 -08:00
Alexander Reelsen eaf67f8bc2 Tests: Replace YAML tests with ESTestRestCase to be able to wait for … (elastic/x-pack-elasticsearch#3252)
* Tests: Replace YAML tests with ESTestRestCase to be able to wait for events

The YAML tests did not have any possibility to wait for the watches to
be created. A hard ten second timeout was used, that could not be
aborted, by simulating a sleep when waiting for a number of nodes that
never occured in the cluster.

This commit replaces those waiting YAML tests with ESRestTestCases, that
use `assertBusy()` to exit early once the watches have been added. Also
this increases the wait time if needed, as these tests tend to fail on
CI.

relates elastic/x-pack-elasticsearch#3217

Original commit: elastic/x-pack-elasticsearch@74b9945d88
2017-12-07 14:40:08 +01:00
David Kyle e9d9199205 [ML] Special Events (elastic/x-pack-elasticsearch#2930)
* Add Special Event

* Add special events to update process

* Add time condition and skip rule actions.

* Update special events

* Address review comments

Original commit: elastic/x-pack-elasticsearch@80500ded76
2017-12-07 11:44:12 +00:00
Costin Leau 432f10e736 SQL: Introduce LiteralAttribute for values that are not extracted from ES (elastic/x-pack-elasticsearch#3209)
This is mainly a promotion of Literal to Attribute to better handle folding expressions from extracted queries

Original commit: elastic/x-pack-elasticsearch@c3bb48bb61
2017-12-06 19:35:34 +02:00
Costin Leau aa9199f5d1 SQL: Push limit down to GroupByColumn aggs (elastic/x-pack-elasticsearch#3200)
Limit operators if available are pushed into `GROUP BY`

Original commit: elastic/x-pack-elasticsearch@e5f82df810
2017-12-06 19:29:48 +02:00
Igor Motov 6839f99ed0 SQL: Switch CLI to core-cli's Command and make it testable (elastic/x-pack-elasticsearch#3232)
Switches CLI to use the standard Elasticsearch Command and refactors it to be more testable. It doesn't change any cli functionality except using the bright color while displaying query results.

relates elastic/x-pack-elasticsearch#2881, elastic/x-pack-elasticsearch#3203, elastic/x-pack-elasticsearch#2990

Original commit: elastic/x-pack-elasticsearch@841f306d50
2017-12-06 11:27:51 -05:00
javanna 9664363575 remove Catalog abstraction and rename its package from catalog to index
Given that the Catalog was only ever used to hold a single index, the corresponding abstraction can be removed in favour of the abstraction that it holds, namely `GetIndexResult`.

Original commit: elastic/x-pack-elasticsearch@6932db642c
2017-12-06 13:58:17 +01:00
javanna 90aee54251 Trim down the Catalog implementations to a single one
Catalog is now a final class rather than an interface with different implementations.

relates elastic/x-pack-elasticsearch#3179

Original commit: elastic/x-pack-elasticsearch@4cc927e113
2017-12-06 13:58:17 +01:00
javanna 8c61cabe87 remove audit logging changes added for delayed actions that are now removed
This commit reverts part of elastic/x-pack-elasticsearch#2210

Original commit: elastic/x-pack-elasticsearch@75fda79851
2017-12-06 13:58:17 +01:00
javanna 08950ff491 Remove security filter, replaced by get index api call which returns filtered mappings
Given that we get now filtered mappings directly from the get index API (in case security is configured with FLS), we don't need the security filter nor the filtered catalog. That means we can remove the delayed action support also from AuthorizationService and rather make SQLAction a composite action like others. It will be authorized as an action, but its indices won't be checked while that will happen with its inner actions (get index and search) which need to be properly authorized.

Also, SQLGetIndicesAction is not needed anymore, as its purpose was to retrieve the indices access resolver put in the context by the security plugin for delayed actions, which are not supported anymore.

This commit kind of reverts elastic/x-pack-elasticsearch#2162, as it is now possible to integrate with security out-of-the-box

relates elastic/x-pack-elasticsearch#2934

Original commit: elastic/x-pack-elasticsearch@64d5044426
2017-12-06 13:58:17 +01:00