Commit Graph

778 Commits

Author SHA1 Message Date
Nik Everett c53e1f4b1c SQL: More tests and docs for permissions (elastic/x-pack-elasticsearch#3679)
Adds tests for what works and what doesn't when we're missing some of
SQL's required permissions.

Adds required permissions to the documentation of each SQL access
method.

relates elastic/x-pack-elasticsearch#3552

Original commit: elastic/x-pack-elasticsearch@971dabb3b4
2018-02-01 17:20:44 -05:00
Martijn van Groningen ee834026d1 Merge changes related to moving persistent tasks to elasticsearch repo into master branch.
Original commit: elastic/x-pack-elasticsearch@fb61e4853c
2018-02-01 15:31:57 +01:00
Lee Hinman 1b36133988 Merge remote-tracking branch 'origin/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@4176773659
2018-01-31 10:49:25 -07:00
Martijn van Groningen d2e7cee41d fixed compile error
Original commit: elastic/x-pack-elasticsearch@ff02a3244c
2018-01-31 15:38:14 +01:00
Dimitris Athanasiou ed11dad855 [ML] Expand detector function shortcuts (elastic/x-pack-elasticsearch#3789)
relates elastic/x-pack-elasticsearch#3162


Original commit: elastic/x-pack-elasticsearch@a3512c6693
2018-01-31 13:11:32 +00:00
Simon Willnauer 570411c2dc Remove all tribe related code, comments and documentation (elastic/x-pack-elasticsearch#3784)
Relates to elastic/elasticsearch#28443

Original commit: elastic/x-pack-elasticsearch@5c4e7fccc7
2018-01-30 20:40:46 +01:00
Nik Everett 6631fe0376 Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@ec3a82494e
2018-01-30 12:50:27 -05:00
Andy Bristol e45b738b4c [TEST] remove check for extensions executables (elastic/x-pack-elasticsearch#3778)
These executables have been removed because the old plugin
infrastructure has been replaced with SPI

Follow up from elastic/x-pack-elasticsearch#3734

Original commit: elastic/x-pack-elasticsearch@89dd1ab3b1
2018-01-30 07:42:44 -08:00
jaymode 0f6ca4716b Test: increase time to wait for the tribe node to start
See elastic/x-pack-elasticsearch#3754

Original commit: elastic/x-pack-elasticsearch@b1fb53ce8e
2018-01-30 08:13:19 -07:00
David Kyle 6c849534da [ML][TEST] Clean up jobs after SE test
Original commit: elastic/x-pack-elasticsearch@c71aafeb06
2018-01-30 14:34:36 +00:00
Tim Vernum 427da8de8b Switch SAML QA test to use a standalone cluster (elastic/x-pack-elasticsearch#3743)
Introduce a healthy dose of reality into the SAML integration tests.
Switches the test class from *Tests to *IT, and updates the build to run it during integTest

Original commit: elastic/x-pack-elasticsearch@edd2538e5f
2018-01-30 23:03:04 +11:00
David Kyle 10fae5e58f [ML][TEST] Scheduled events integration test (elastic/x-pack-elasticsearch#3555)
* Scheduled events integration test

Original commit: elastic/x-pack-elasticsearch@8181645fe5
2018-01-30 10:38:13 +00:00
Lee Hinman a789944b8d Merge remote-tracking branch 'origin/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@f96f80f0d0
2018-01-29 15:42:01 -07:00
Simon Willnauer 32d4e7f9d0 [TEST] Use `ensureYellowAndNoInitializingShards` in monitoring test (elastic/x-pack-elasticsearch#3768)
Monitoring creates indices concurrently to the test execution. In that
case ensureYellow might not be enough and will cause test-failures when shards
are not active etc. This change uses a new method added in core to ensure shards
are not initializing anymore.

relates elastic/x-pack-elasticsearch#2672
Relates to elastic/elasticsearch#28416

Original commit: elastic/x-pack-elasticsearch@661e87f2ee
2018-01-29 20:46:35 +01:00
Costin Leau 07658cc04f SQL: replace JDBC transport meta columns with SQL call (elastic/x-pack-elasticsearch#3740)
Replace meta column endpoint with SYS COLUMNS command

Original commit: elastic/x-pack-elasticsearch@819874bc5b
2018-01-29 19:18:24 +02:00
David Kyle b6bab14fd4 Revert "Mute ML rolling upgrade tests. Awaits fix elastic/x-pack-elasticsearch#1760"
This reverts commit elastic/x-pack-elasticsearch@deaf060818.

Original commit: elastic/x-pack-elasticsearch@2eece7d27e
2018-01-29 13:24:07 +00:00
Dimitris Athanasiou 664fcccb8b [ML][TEST] Add integ test for categorical rule (elastic/x-pack-elasticsearch#3739)
Original commit: elastic/x-pack-elasticsearch@975a10f810
2018-01-29 11:45:04 +00:00
Costin Leau e929d16d76 SQL: Replace ListTable transport with SQL command (elastic/x-pack-elasticsearch#3675)
Introduce system commands as alternative to meta HTTP endpoints
Pass in cluster name
Use 'BASE TABLE' instead of 'INDEX' when describing a table to stick
with the SQL terminology

Original commit: elastic/x-pack-elasticsearch@600312b8f7
2018-01-27 11:54:19 +02:00
Ryan Ernst 5a86450df7 Build: Replace references to x-pack-elasticsearch paths with helper methods (elastic/x-pack-elasticsearch#3748)
In order to more easily integrate xpack once it moves into the
elasticsearch repo, references to the existing x-pack-elasticsearch need
to be reduced. This commit introduces a few helper "methods" available
to any project within xpack (through gradle project extension
properties, as closures). All refeerences to project paths now use these
helper methods, except for those pertaining to bwc, which will be
handled in a followup.

Original commit: elastic/x-pack-elasticsearch@850668744c
2018-01-26 21:48:30 -08:00
Dimitrios Athanasiou b2f0745c7b [ML][TEST] Use condition_type instead of type in rule bwc test
Original commit: elastic/x-pack-elasticsearch@f49fe28b4e
2018-01-26 18:46:11 +00:00
Dimitrios Athanasiou 621f4f20b0 [ML][TEST] Add BWC tests for rules syntax change
Original commit: elastic/x-pack-elasticsearch@805d390417
2018-01-26 17:37:08 +00:00
Simon Willnauer 730e7075ab Remove XPackExtension in favor of SecurityExtensions (elastic/x-pack-elasticsearch#3734)
This change removes the XPackExtension mechanism in favor of
SecurityExtension that can be loaded via SPI and doesn't need
another (duplicate) plugin infrastructure

Original commit: elastic/x-pack-elasticsearch@f39e62a040
2018-01-26 16:14:11 +01:00
David Kyle 2356830782 [ML][TEST] Disable monitoring for ML QA tests (elastic/x-pack-elasticsearch#3751)
Original commit: elastic/x-pack-elasticsearch@4c3192e964
2018-01-26 10:10:52 +00:00
Ryan Ernst 3e24ea3ed2 Test: Add internalClusterTest to xpack modules (elastic/x-pack-elasticsearch#3730)
This commit reenables running ITs in xpack by adding an internalClusterTest to xpack modules that contain ESIntegTestCase tests. The new task allows us to run these independently of rest integ tests, which are disabled for xpack modules because installing the bundled plugins directly is not quite the same as installing via the meta plugin. Some tests (ML) are moved to their own qa module to accommodate the need for a real cluster. A couple tests (monitoring and upgrade) have been marked as AwaitsFix.

Commits that have been folded into this commit:
* Move ML IT tests to qa/ml-native-tests
* Add internalClusterTest task and disable rest integ tests for xpack
modules. Also tweak ML tests and get upgrade tests working
* Adding the keystore and security back to the ml native tests
* Fixing native integ test
* Fix last ML test, add awaits fix to monitoring and upgrade tests
* cleanup PR
* fix checkstyle

Original commit: elastic/x-pack-elasticsearch@3c0ed6fd3b
2018-01-25 10:38:34 +01:00
Tanguy Leroux 7c90f874d7 Update x-pack packaging tests to work with meta plugins (elastic/x-pack-elasticsearch#3683)
relates elastic/x-pack-elasticsearch#3652

Original commit: elastic/x-pack-elasticsearch@e177b742f9
2018-01-25 10:20:52 +01:00
jaymode b456885b39 Test: increase time to wait for tribe tests with security
There have been some failures in CI due to the tribe tests with security taking too long for the
cluster to form and the expected number of node to join. In the thread dumps of the failure, it
can be seen that a node is still initializing and is in a method that could take a decent amount
of time on slow machines (TokenService#computeSecretKey). This commit increases the wait time to
allow for the node to startup and join.

Original commit: elastic/x-pack-elasticsearch@2bf4c96d8f
2018-01-24 16:30:02 -07:00
jaymode 2c46002c00 Test: get the rest-api-spec from the correct location
Original commit: elastic/x-pack-elasticsearch@a3f3c45213
2018-01-24 10:39:43 -07:00
Costin Leau 3f4c789993 Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@348f0468b0
2018-01-24 19:19:51 +02:00
jaymode 8059099697 Test: re-enable the rolling-upgrade tests after the refresh token backport
Original commit: elastic/x-pack-elasticsearch@a39736a4c0
2018-01-24 10:09:47 -07:00
Igor Motov de15f7123b Merge remote-tracking branch 'elastic/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@bbabfb1328
2018-01-23 15:41:59 -05:00
Jason Tedor c0790d6a49 Move x-pack-core to core package (elastic/x-pack-elasticsearch#3678)
This commit moves the source file in x-pack-core to a org.elasticsearch.xpack.core package. This is to prevent issues where we have compile-time success reaching through packages that will cross module boundaries at runtime (due to being in different classloaders). By moving these to a separate package, we have compile-time safety. Follow-ups can consider build time checking that only this package is defined in x-pack-core, or sealing x-pack-core until modules arrive for us.

Original commit: elastic/x-pack-elasticsearch@232e156e0e
2018-01-23 12:43:58 -06:00
Igor Motov 2be9cabbe9 Merge remote-tracking branch 'elastic/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@d6451b0f6b
2018-01-23 12:53:13 -05:00
Nik Everett 52f7ba8c5d SQL: Docs for basic aggregations
Adds documentation for basic aggregate functions supported by
Elasticsearch SQL.

Relates to elastic/x-pack-elasticsearch#2898

Original commit: elastic/x-pack-elasticsearch@ddc71165f2
2018-01-23 11:32:21 -05:00
Simon Willnauer 63c0e288af Ensure we protect Collections obtained from scripts from self-referencing (elastic/x-pack-elasticsearch#3681)
Self referencing maps can cause SOE if they are iterated ie. in their toString methods. This chance adds some protected to the usage of those collections.
see elastic/elasticsearch#28335

Original commit: elastic/x-pack-elasticsearch@c4f1089c74
2018-01-23 17:02:37 +01:00
Nik Everett d2ea36416b SQL: Expose WEEK function and document datetime functions (elastic/x-pack-elasticsearch#3680)
Adds documentation for all of the date time functions using the new
cli-like format extracted from the csv spec. In the process of doing
this I noticed that the `WEEK` function isn't exposed as a function.
This exposes it for consistency.

Relates to elastic/x-pack-elasticsearch#2898

Original commit: elastic/x-pack-elasticsearch@0459b24cb9
2018-01-23 07:11:43 -05:00
Simon Willnauer c70c1c7e98 Expose XPackExtensions via SPI (elastic/x-pack-elasticsearch#3530)
This change adds SPI loading for XPackExtensions that allows to extend
XPack via an ordinary plugin. This can co-exist with the existin
extension mechanism for the time being.

Original commit: elastic/x-pack-elasticsearch@bf02b56dee
2018-01-23 13:05:39 +01:00
Igor Motov 6330209e7d Merge remote-tracking branch 'elastic/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@95cbd93803
2018-01-22 16:03:56 -05:00
Dimitris Athanasiou cc432aee34 [ML] Check calendar exists before removing job (elastic/x-pack-elasticsearch#3661)
Also, removes check for whether a job-to-remove exists
and replaces it with a check of whether a job-to-remove
is already present in the calendar. This allows to
remove a job that may no longer exists and it improves
feedback for the case that an existing job is removed from
a calendar that doesn't contain it.

relates elastic/x-pack-elasticsearch#3620

Original commit: elastic/x-pack-elasticsearch@3ea39be1b6
2018-01-22 13:49:38 +00:00
Dimitris Athanasiou f3282b559f [ML] Check calendar exists before deleting event (elastic/x-pack-elasticsearch#3659)
Solves the event part of elastic/x-pack-elasticsearch#3620

Original commit: elastic/x-pack-elasticsearch@c79ca85c6e
2018-01-22 13:02:42 +00:00
Jason Tedor 928b6a6e04 Fix compilation in audit-tests
This commit fixes compilation in audit-tests which was broken after some
stray classes were moved out of plugin into specific plugin directories.

Original commit: elastic/x-pack-elasticsearch@8bfd8cfceb
2018-01-22 06:00:23 -05:00
Ryan Ernst ac44ee8257 Build: Move test files from plugin/src to appropriate other projects (elastic/x-pack-elasticsearch#3653)
This commit moves mostly security tests that were left in plugin/src. It
also moves the CompositeTestingXPackPlugin into tribe license tests qa
project, which was the only remaining use. This class needs to be
removed, as it has unavoidable problems with dependency conflicts (ie
guava) between security and watcher, which it pulls in both into the
test classpath.

Original commit: elastic/x-pack-elasticsearch@756209e010
2018-01-21 14:08:47 -05:00
Jason Tedor 377bb53cca Fix format of third party audit in saml-idp-tests
The formatting here was off since it contains tabs and tabs are evil.

Original commit: elastic/x-pack-elasticsearch@fadbb02157
2018-01-20 19:53:26 -05:00
Tim Vernum 47213f5675 [Security] Add SAML authentication support (elastic/x-pack-elasticsearch#3646)
Introduces:

- SAML Realm
- REST & Transport actions to support SAML single signon / signout
- Tests for above
- More XML than you ever wanted to see.

Original commit: elastic/x-pack-elasticsearch@b0fe7bb652
2018-01-21 08:43:00 +10:00
Ryan Ernst 0bc496ae04 add pattern excludes for binary files
Original commit: elastic/x-pack-elasticsearch@f1077e35f8
2018-01-19 22:05:01 -08:00
Michael Basnight f3ec4a5208 Split up xpack plugins into their own modules (elastic/x-pack-elasticsearch#3643)
Thanks to some great work by a bunch of amazing people, the chuck norris xpack split is a go!

Original commit: elastic/x-pack-elasticsearch@dad98e28f4
2018-01-19 23:30:17 -06:00
Nik Everett 51a6285ba1 SQL: Allow CSV tests to look more like the CLI (elastic/x-pack-elasticsearch#3640)
This allows CSV tests to include a line between the header and the
values that is ignored by the test framework. This optional line can be
added to the tests to make them a little easier to read which is useful
when they are included in the documentation. As a side effect they also
closely mimick the output of the CLI. To the point where you can copy
directly from the CLI and paste into the CSV tests.

Example:
```
constantYear
// tag::year
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
     year
---------------
2018
// end::year
;
```

This can be extracted with a construct like this in the docs:
```
["source","sql",subs="attributes,callouts,macros"]
--------------------------------------------------
include-tagged::{sql-specs}/datetime.csv-spec[year]
--------------------------------------------------
```

Which makes documentation that looks like this:
```
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
     year
---------------
2018
```

Which is fairly nice.

Original commit: elastic/x-pack-elasticsearch@8c10b5cb10
2018-01-19 16:24:42 -05:00
Nik Everett b165f1c71e SQL: Fix constant folding of datetime functions (elastic/x-pack-elasticsearch#3637)
I went to write some docs for datetime functions that look like:
```
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;

  year
2018

```
because I figured they'd be pretty easy to read because they didn't
require any knowledge of a data set. But it turns out that constant
folding doesn't work properly for date time functions because they don't
actually apply the extraction.

Original commit: elastic/x-pack-elasticsearch@aa9c66b2c7
2018-01-19 15:11:37 -05:00
Dimitris Athanasiou 21f692c02b [ML] Further validate calendar_id and add calendar description (elastic/x-pack-elasticsearch#3624)
relates elastic/x-pack-elasticsearch#3595

Original commit: elastic/x-pack-elasticsearch@fade977361
2018-01-19 10:44:39 +00:00
Lee Hinman 5818986775 Merge remote-tracking branch 'origin/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@8475419a00
2018-01-18 16:51:56 -07:00
Nik Everett b9dbb6d204 SQL: Run fewer tests against multiple nodes (elastic/x-pack-elasticsearch#3625)
Rather than run every SQL test against multiple nodes we instead run a
subset of the tests simply for speed. The subset is both small but
exercises the CLI, JDBC, and REST interface and a few special "these
might be different in against multiple nodes" cases.

This drops the run time from of `gradle clean check` of these tests from
```
BUILD SUCCESSFUL in 4m 49s
```

To:
```
BUILD SUCCESSFUL in 2m 49s
```

Original commit: elastic/x-pack-elasticsearch@dcd99bcd82
2018-01-18 15:59:40 -05:00
Costin Leau e48589aa65 Fix CLI test
Original commit: elastic/x-pack-elasticsearch@8a31108b09
2018-01-18 22:33:47 +02:00
Nik Everett 9b6f417df6 SQL: Move the jdbc directory into plugin/sql (elastic/x-pack-elasticsearch#3617)
This is the next step in removing the top level `sql` directory.

Related to elastic/x-pack-elasticsearch#3363

Original commit: elastic/x-pack-elasticsearch@948c3e3583
2018-01-18 09:33:16 -05:00
Costin Leau 1dd98eab83 SQL: Fix show tables as admin (elastic/x-pack-elasticsearch#3597)
Fix show tables as JDBC with security enabled
This commit fixes the test SqlSecurityTestCase.testShowTablesWorksAsAdmin to
skip over any indices/aliases that exist starting with `.security`.
Use value comparison instead of the result sets
Fix an offset bug while at it (columns start at 1 not 0)
Resolves elastic/x-pack-elasticsearch#3423

Original commit: elastic/x-pack-elasticsearch@6fffda6070
2018-01-18 16:13:01 +02:00
Nik Everett 753d21f3c6 SQL: Move CLI into plugin/sql directory (elastic/x-pack-elasticsearch#3606)
This is the next step in removing the top level sql directory.

I named the directory `sql-cli` instead of `cli` because that puts it at
the maven coordinates `org.elasticsearch.plugin:sql-cli` instead of
`org.elasticsearch.plugin:cli`.

Relates to elastic/x-pack-elasticsearch#3363

Original commit: elastic/x-pack-elasticsearch@d41a57a136
2018-01-18 08:39:02 -05:00
Nik Everett 310d1d2302 SQL: Move `sql:server` to `plugin:sql` (elastic/x-pack-elasticsearch#3604)
This moves SQL's server project into `plugin:sql` without modifying how the integration is performed. I know that it is not correct with regards to the x-pack modularization but I think it is a good first step.

Original commit: elastic/x-pack-elasticsearch@2f40d02e4d
2018-01-17 15:48:58 -05:00
Costin Leau 41c1c5fdd1 SQL: close ResultSet after asserting (elastic/x-pack-elasticsearch#3601)
The on-closing Connection is closed so there's no leak however closing
the ResultSet is good practice and tests another piece of code as well

Original commit: elastic/x-pack-elasticsearch@3a9cee70a3
2018-01-17 22:14:07 +02:00
Nik Everett d6e292087b SQL: Drop embedded qa mode (elastic/x-pack-elasticsearch#3602)
It will conflict with moving the `sql:server` project to `plugin:sql`
and we're fairly sure we need to rework or remove it anyway.

relates elastic/x-pack-elasticsearch#3557

Original commit: elastic/x-pack-elasticsearch@763072c182
2018-01-17 15:04:14 -05:00
Jay Modi 60d4b7e53e Add the ability to refresh tokens obtained via the API (elastic/x-pack-elasticsearch#3468)
This commit adds the ability to refresh tokens that have been obtained by the API using a refresh
token. Refresh tokens are one time use tokens that are valid for 24 hours. The tokens may be used
to get a new access and refresh token if the refresh token has not been invalidated or
already refreshed.

relates elastic/x-pack-elasticsearch#2595

Original commit: elastic/x-pack-elasticsearch@23435eb815
2018-01-17 12:18:44 -07:00
Igor Motov c3b82e5ee1 SQL: Remove test-utils project (elastic/x-pack-elasticsearch#3583)
Makes Sql Translate Action consistent with other SQL Actions and removes test-utils project

Follow up for elastic/x-pack-elasticsearch#3543

Original commit: elastic/x-pack-elasticsearch@8ff2148d67
2018-01-17 12:18:25 -05:00
Lee Hinman f53a19374b SQL: Wrap coercion in JdbcResultSet to throw SQLException (elastic/x-pack-elasticsearch#3559)
* SQL: Wrap coercion in JdbcResultSet to throw SQLException

This catches the `ClassCastException` that could be thrown when retrieving data
from a result set, instead converting it into a `SQLException`.

Resolves elastic/x-pack-elasticsearch#3207

* Add simple test for incorrect coercion

Original commit: elastic/x-pack-elasticsearch@5480a48d95
2018-01-16 14:39:20 -07:00
Igor Motov 9401569195 SQL: Adapt to compile and runtime Java home distinction
Relates elastic/x-pack-elasticsearch#3477

Original commit: elastic/x-pack-elasticsearch@5791617988
2018-01-16 14:53:49 -05:00
Igor Motov 35b45e9a14 Merge remote-tracking branch 'elastic/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@5322205268
2018-01-16 14:43:54 -05:00
Jason Tedor 8a0a14b9a3 Adapt to compile and runtime Java home distinction
We now separate the compiler Java home from the time runtime Java home
(the one that is used to compile class files versus the one that is used
to run tests). This commit adapts x-pack-elasticsearch to this change.

Relates elastic/x-pack-elasticsearch#3477

Original commit: elastic/x-pack-elasticsearch@bdb096e21c
2018-01-16 13:45:42 -05:00
Igor Motov 12d4f10faa SQL: Switch JDBC to REST protocol (elastic/x-pack-elasticsearch#3543)
Replaces binary serialization protocol in JDBC driver with the common REST protocol.

relates elastic/x-pack-elasticsearch#3419

Original commit: elastic/x-pack-elasticsearch@532c106658
2018-01-16 13:26:06 -05:00
Costin Leau 448511e093 Merge remote-tracking branch 'remotes/upstream/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@2c78f17486
2018-01-16 17:11:37 +02:00
Costin Leau 7d3c6a778f SQL: update support for nested docs (elastic/x-pack-elasticsearch#3546)
* SQL: update support for nested docs

Expand sample data with nested docs
Optimize source generation for nested parents
Nested objects fields are specified in inner-hits and don't need to be
specified at the top-level query. Further more disable parent source if
not needed.
ComputingFieldRef needs to return the hitName of its wrapped children
otherwise nested values cannot be extracted
Disable GROUP BY/HAVING on nested fields
Update FieldAttribute requirements in the random tests

Original commit: elastic/x-pack-elasticsearch@e44951b5f6
2018-01-16 16:59:22 +02:00
Dimitris Athanasiou 2350b85507 [ML] Improve error on get calendar when ml-meta index is missing (elastic/x-pack-elasticsearch#3576)
Relates elastic/x-pack-elasticsearch#3016

Original commit: elastic/x-pack-elasticsearch@bc60a0cb72
2018-01-16 11:00:07 +00:00
Dimitris Athanasiou 6c6dcafd0e [ML] Improve error message when post-calendar-events-request is empty (elastic/x-pack-elasticsearch#3572)
Relates elastic/x-pack-elasticsearch#3016

Original commit: elastic/x-pack-elasticsearch@3d25ed4723
2018-01-15 18:13:50 +00:00
Alexander Reelsen 57d887c9df Tests: Add noop test to prevent test failures
Follow up for elastic/x-pack-elasticsearch@a15338415a

Original commit: elastic/x-pack-elasticsearch@c0a46a5977
2018-01-15 18:06:54 +01:00
Alexander Reelsen d4cddc12d0 Tests: Add @AwaitsFix annotation to flaky tests
These contribute to unstable builds, and should be reenabled on a
case by case base and worked on.

Original commit: elastic/x-pack-elasticsearch@a15338415a
2018-01-15 10:28:05 +01:00
Igor Motov 70cea58262 Merge remote-tracking branch 'elastic/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@9ac41d008e
2018-01-12 15:35:56 -05:00
Jay Modi cec90f452a IndexLifecycleManager checks for index existense and up to date mappings (elastic/x-pack-elasticsearch#3515)
This change makes the IndexLifecycleManager check for both index existence and up to date mappings
on the index prior to executing the provided runnable. Doing this provides a mechanism to make
non-breaking mapping updates to the security index in minor versions.

relates elastic/x-pack-elasticsearch#3462

Original commit: elastic/x-pack-elasticsearch@80f05d83b4
2018-01-12 12:45:52 -07:00
Igor Motov 915ed35e27 Merge remote-tracking branch 'elastic/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@02fddccd84
2018-01-11 13:41:40 -05:00
David Roberts a386b5727e [ML] Increase tokenization flexibility for categorization (elastic/x-pack-elasticsearch#3372)
By moving tokenization for categorization to Java we give users access to considerably more options for tokenizing their log messages prior to using ML to categorize them. Now all Elasticsearch analyzer functionality is available, which opens up the possibility to sensibly categorize non-English log messages.

Relates elastic/machine-learning-cpp#491

Original commit: elastic/x-pack-elasticsearch@5d61b67614
2018-01-11 15:40:13 +00:00
Chris Earle e74f90eba0 [Monitoring] Add Cluster Alert for Nodes Changed (elastic/x-pack-elasticsearch#3504)
This creates a cluster alert that triggers whenever a node is:

- Added
- Removed
- Restarted (aka Removed and Added between collections)

Unlike all previous cluster alerts, this cluster is always instantly resolved because there is no way to otherwise resolve it later (especially if nodes are replaced).

This will require a small change to the UI so that it does not simply ignore resolved alerts.

Original commit: elastic/x-pack-elasticsearch@6340bf7960
2018-01-10 21:29:00 -05:00
Igor Motov ce81a34467 SQL: Add jdbc mode to REST API (elastic/x-pack-elasticsearch#3499)
Adds a mode parameter to all SQL-related requests. The mode parameter is used for license checks as well as to define the response content. For now only two modes are supported plain (default) and jdbc. We will add other modes in the future as we add more clients.

Relates elastic/x-pack-elasticsearch#3419 

Original commit: elastic/x-pack-elasticsearch@b49ca38d4b
2018-01-10 09:41:36 -05:00
Lee Hinman 0865063740 SQL: Fix multi-line CLI commands to join correctly (elastic/x-pack-elasticsearch#3516)
Previously multi-line CLI SQL statements were joined, but the space command was
missing, so a command like:

```
sql> SHOW
   | functions;
```

Would incorrectly parse as "showfunctions" and throw an error.

This fixes the behavior and adds a test for multi-line commands.

Resolves elastic/x-pack-elasticsearch#3410

Original commit: elastic/x-pack-elasticsearch@3870924ccd
2018-01-09 16:55:30 -07:00
Costin Leau 8257d8d76a SQL: Extend grammar to allow wildcard in the grammar without quotes (elastic/x-pack-elasticsearch#3501)
Tweak the grammar to differentiate between table and normal identifier.
The table one allows wildcard while the normal one (for fields) does not.

Original commit: elastic/x-pack-elasticsearch@a714e950db
2018-01-09 19:30:50 +02:00
Costin Leau 25a00a3b55 SQL: Check null in processor (elastic/x-pack-elasticsearch#3494)
Make Processors resilient to NULL values
Check null only in functions not constants

Original commit: elastic/x-pack-elasticsearch@dd8bd16d49
2018-01-09 16:18:16 +02:00
Alexander Reelsen b2493aad8d Tests: Added trace logging for more debug information
Relates elastic/x-pack-elasticsearch#3430

Original commit: elastic/x-pack-elasticsearch@67aeebcc4a
2018-01-08 11:49:20 +01:00
Costin Leau aa85496090 Make debug tests abstract again
Original commit: elastic/x-pack-elasticsearch@1c9b60d556
2018-01-07 00:12:44 +02:00
Costin Leau ba81321d0d SQL: Support aliases and indices pattern (elastic/x-pack-elasticsearch#3438)
Add support for aliases and indices pattern
Enhance ShowTable info to differentiate between aliases and indices
Add regex filtering of index names
Handle security exceptions (in case of no privileges or no matching)

Original commit: elastic/x-pack-elasticsearch@91e3674ca7
2018-01-06 23:08:16 +02:00
Igor Motov db1693ea20 Merge remote-tracking branch 'elastic/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@7029b1f881
2018-01-03 15:16:21 -05:00
Alexander Reelsen e9abf18089 Tests: Increase logging to see requests
This test case fails on non-reproducibly on CI, so this adds
request/response logging to make sure all the requests were actually
successful.

Relates elastic/x-pack-elasticsearch#3340

Original commit: elastic/x-pack-elasticsearch@9e6c359f8c
2018-01-02 17:38:52 +01:00
Igor Motov 9f71100bac SQL: Fix CLI tests with security on Windows
We test on Windows with the terminal in unix mode, so we need to make sure we send `\n` instead of `\r\n`. In this case the extra `\n` after password was causing an extra new line in tests breaking them on Windows.

Original commit: elastic/x-pack-elasticsearch@721ba32bd6
2017-12-28 21:40:03 -05:00
Igor Motov 178d6ca6b3 SQL: Move shared REST client methods to shared-client (elastic/x-pack-elasticsearch#3428)
* SQL: Move shared REST client methods to shared-client

This commit is a preliminary step for moving JDBC to the REST client. It extracts the common REST clients from CLI and moves it to shared-client. This will allow us to move to the 5 project setup: rest-proto, shared-client, server, jdbc, cli with the following dependencies:

server <-- rest-proto
shared-client <-- rest-proto
jdbc <-- shared-client
cli <-- shared-client

Relates to elastic/x-pack-elasticsearch#3419

Original commit: elastic/x-pack-elasticsearch@2e6a134de0
2017-12-28 11:10:18 -05:00
Igor Motov 292506526e Merge remote-tracking branch 'elastic/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@79e7b1b953
2017-12-27 21:46:00 -05:00
Michael Basnight 220aa734ee Migrate actions to plugin core (elastic/x-pack-elasticsearch#3424)
This commit hacks up the xpack repo in such a way that it moves all
actions that the transport client uses to plugin core. It also moves
any classes that those actions use to plugin core, with a few
exceptions. I tried to split up any classes that pulled in server side
logic into the client, but that was not always 100% possible. Consider
this commit a guide, and since I do not know the codebase for each
plugin, consider it a best guess for what should be moved. A few other
things were merged in below.

This commit extracts the parts of Condition and AlwaysCondition that are
needed by classes that will be moved into the client package. The only
odd thing here is that since there are two parent classes (Condition and
AlwaysCondition) it was not possible to make AlwaysConditionInteral
extend from AlwaysCondition and still parse a ConditionInternal object,
which is what the ConditionFactory expects.

This commit removes the use of internal users in the User class, and
instead moves them to Authentication, where they are used.

[insert obligatory chuck norris karate image here]

ref elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@42d0b72209
2017-12-27 11:31:16 -05:00
Igor Motov 3892da7a3d SQL: Suppress testShowTablesWorksAsAdmin() tests
These tests are relying on existence of an internal index that might or might not be present when we run this test.

Tracked by elastic/x-pack-elasticsearch#3423

Original commit: elastic/x-pack-elasticsearch@26a7795759
2017-12-26 12:53:21 -05:00
Costin Leau 5a25213db9 Update security tests
Original commit: elastic/x-pack-elasticsearch@732a4c5850
2017-12-23 15:44:27 +02:00
Costin Leau b437341ddc Remove old class
Original commit: elastic/x-pack-elasticsearch@e075c3b972
2017-12-23 13:35:42 +02:00
Costin Leau 368099dcbd SQL: Add support for object/inner/dotted fields (elastic/x-pack-elasticsearch#3368)
* SQL: Add support for object/inner/dotted fields

Improve validation when selecting compound fields
Improve fallback to exact field for text fields
Consolidate various field attributes into one
Consolidate various field HitExtractors into one
Improve grammar for column declaration
Update response of embedded server
Improve index validation to allow use of internal indices
Add clean-up of scroll in case of server errors
By default thrown an exception on multi-valued fields

Original commit: elastic/x-pack-elasticsearch@45b73fe0dc
2017-12-23 13:19:54 +02:00
Igor Motov e18c7e92fa SQL: Switch CLI to REST protocol (elastic/x-pack-elasticsearch#3342)
Removes custom binary protocol used for CLI communication and switches CLI to REST.

relates elastic/x-pack-elasticsearch#3063

Original commit: elastic/x-pack-elasticsearch@b42bdcdc35
2017-12-22 20:42:13 -10:00
Nik Everett 1cf9d6e3f3 SQL: Document a few functions (elastic/x-pack-elasticsearch#3390)
* Starts to build the list of supported functions, adding links to
wikipedia when there is any doubt what the functions mean.
* Extracts an example of using the function from the test suite.
* Explicitly calls out how we round (half up) because there are
lots of ways to round.

Original commit: elastic/x-pack-elasticsearch@5fb64ba869
2017-12-20 17:42:29 -05:00
Lee Hinman c26f039207 SQL: Add all QUERY() query options (elastic/x-pack-elasticsearch#3389)
This adds support for (almost) all of the options that the `query_string` query
supports.

Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `QUERY()` query.

Relates to elastic/x-pack-elasticsearch#3361

Original commit: elastic/x-pack-elasticsearch@da8b29b53c
2017-12-20 15:05:40 -07:00
Lee Hinman 5b47c67dec SQL: Add all MATCH() query options (elastic/x-pack-elasticsearch#3387)
* SQL: Add all MATCH() query options

This adds support for (almost) all of the options that the `match` query
supports.

Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `MATCH()` query.

Relates to elastic/x-pack-elasticsearch#3361

* Add getters required for Node usage

Original commit: elastic/x-pack-elasticsearch@144e2bec02
2017-12-20 12:35:32 -07:00
Nik Everett 42d1c62d03 SQL: Nicer failure message one EXISTS (elastic/x-pack-elasticsearch#3384)
We don't support the SQL `EXISTS` keyword. It looks like:
```
SELECT * FROM test WHERE EXISTS (SELECT * FROM foo WHERE test.id =
foo.id)
```

It is basically a `JOIN` that doesn't return columns. Since we don't
support `JOIN`, we don't support `EXISTS`.

This PR improves the error message from "unresolved blah blah blah" to
"EXISTS is not yet supported".

relates elastic/x-pack-elasticsearch#3176

Original commit: elastic/x-pack-elasticsearch@548d57c8c1
2017-12-20 12:04:46 -05:00
Lee Hinman 38cee807b9 SQL: Parse all multi_match options from request (elastic/x-pack-elasticsearch#3361)
* Parse all multi_match options from request

Rather than hardcoding predicates that require more methods for each newly added
options, this encapsulates the state into a collection of appliers that can
accumulate what's needed in the QueryBuilder itself.

For example, the following:

```json
GET /_xpack/sql/translate
{
  "query": "SELECT foo,baz from i WHERE MATCH('baz', 'should clause', 'operator=AND;type=cross_fields')"
}
```

Then generates the following:

```json
{
  "size" : 1000,
  "query" : {
    "multi_match" : {
      "query" : "should clause",
      "fields" : [
        "baz^1.0"
      ],
      "type" : "cross_fields",
      "operator" : "AND",
      "slop" : 0,
      "prefix_length" : 0,
      "max_expansions" : 50,
      "zero_terms_query" : "NONE",
      "auto_generate_synonyms_phrase_query" : true,
      "fuzzy_transpositions" : true,
      "boost" : 1.0
    }
  },
  "_source" : {
    "includes" : [
      "baz"
    ],
    "excludes" : [ ]
  },
  "docvalue_fields" : [
    "foo"
  ]
}
```

And when an invalid field value is used:

```json
GET /_xpack/sql
{
  "query": "SELECT foo,baz from i WHERE MATCH('baz', 'should clause', 'operator=AND;minimum_should_match=potato')"
}
```

We get what ES would usually send back:

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "query_shard_exception",
        "reason" : "failed to create query: {\n  \"multi_match\" : {\n    \"query\" : \"should clause\",\n    \"fields\" : [\n      \"baz^1.0\"\n    ],\n    \"type\" : \"best_fields\",\n    \"operator\" : \"AND\",\n    \"slop\" : 0,\n    \"prefix_length\" : 0,\n    \"max_expansions\" : 50,\n    \"minimum_should_match\" : \"potato\",\n    \"zero_terms_query\" : \"NONE\",\n    \"auto_generate_synonyms_phrase_query\" : true,\n    \"fuzzy_transpositions\" : true,\n    \"boost\" : 1.0\n  }\n}",
        "index_uuid" : "ef3MWf8FTUe2Qjz2FLbhoQ",
        "index" : "i"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "i",
        "node" : "VfG9zfk9TDWdWvEZu0a4Rw",
        "reason" : {
          "type" : "query_shard_exception",
          "reason" : "failed to create query: {\n  \"multi_match\" : {\n    \"query\" : \"should clause\",\n    \"fields\" : [\n      \"baz^1.0\"\n    ],\n    \"type\" : \"best_fields\",\n    \"operator\" : \"AND\",\n    \"slop\" : 0,\n    \"prefix_length\" : 0,\n    \"max_expansions\" : 50,\n    \"minimum_should_match\" : \"potato\",\n    \"zero_terms_query\" : \"NONE\",\n    \"auto_generate_synonyms_phrase_query\" : true,\n    \"fuzzy_transpositions\" : true,\n    \"boost\" : 1.0\n  }\n}",
          "index_uuid" : "ef3MWf8FTUe2Qjz2FLbhoQ",
          "index" : "i",
          "caused_by" : {
            "type" : "number_format_exception",
            "reason" : "For input string: \"potato\""
          }
        }
      }
    ]
  },
  "status" : 400
}
```

It even includes the validation that ES already does for things like `type`:

```json
GET /_xpack/sql
{
  "query": "SELECT foo,baz from i WHERE MATCH('baz', 'should clause', 'operator=AND;type=eggplant')"
}
```

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "failed to parse [multi_match] query type [eggplant]. unknown type."
      }
    ],
    "type" : "parse_exception",
    "reason" : "failed to parse [multi_match] query type [eggplant]. unknown type."
  },
  "status" : 400
}
```

Resolves elastic/x-pack-elasticsearch#3257


Original commit: elastic/x-pack-elasticsearch@59f518af4a
2017-12-20 09:48:05 -07:00
Nik Everett 2c5cfcfae4 SQL: Proper errors on set qualifiers (elastic/x-pack-elasticsearch#3377)
`SELECT DISTINCT foo FROM bar` is not yet implemented and was returning
an "unplanned item" error which isn't useful to users so I replaced it
with `SELECT DISTINCT is not yet supported`.

`SELECT foo, COUNT(*) FROM bar GROUP BY ALL foo` is not supported.
Specifically the `ALL` part. It is fairly esoteric so see [1] for what
it does. Regardless of what it does it is not widely supported and even
Microsoft's SQL Server has deprecated it so we should never support it.
Probably.

[1]: https://technet.microsoft.com/en-us/library/ms175028(v=sql.90).aspx

Related to elastic/x-pack-elasticsearch#3176

Original commit: elastic/x-pack-elasticsearch@56e5ca3009
2017-12-20 09:25:00 -05:00
Nik Everett 7e11a1b388 Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@82985d6481
2017-12-19 13:43:49 -05:00
jaymode f1f1be3927 Test: tests that use security index should not delete template
Tests that rely on the security index and security index template being present should not remove
the template between tests as this can cause test failures. The template upgrade service relies
on cluster state updates to trigger the template being added after a delete, but there is a
scenario where the test will just wait for template that never shows up as there is no cluster
state update in that time. Instead of fighting ourselves, we should just leave the template in
place.

Relates elastic/x-pack-elasticsearch#2915
Relates elastic/x-pack-elasticsearch#2911

Original commit: elastic/x-pack-elasticsearch@3ca4aef0be
2017-12-19 08:24:23 -07:00
Nik Everett 4820bc757e SQL: Implement sorting and retrieving score (elastic/x-pack-elasticsearch#3340)
Accessing `_score` from SQL looks like:
```
--------------------------------------------------
POST /_sql
{
    "query": "SELECT SCORE(), * FROM library WHERE match(name, 'dune') ORDER BY SCORE() DESC"
}
--------------------------------------------------
```

This replaces elastic/x-pack-elasticsearch#3187

relates elastic/x-pack-elasticsearch#2887

Original commit: elastic/x-pack-elasticsearch@fe96348c22
2017-12-18 20:57:50 -05:00
Nik Everett dc69f92b49 SQL: Respond with nice error if there is a JOIN (elastic/x-pack-elasticsearch#3343)
`JOIN`s aren't supported right yet so we should send back a nice 400
level error to the user telling them that.

Also pulls out some common code in `RestSqlTestCase` that I've been
staring at for a while.

Relates to elastic/x-pack-elasticsearch#3176

Original commit: elastic/x-pack-elasticsearch@1c1bd1c355
2017-12-18 16:42:03 -05:00
Lee Hinman a8e9272994 SQL: Add remaining matrix aggregations (elastic/x-pack-elasticsearch#3330)
* Add remaining matrix aggregations

This adds the remaining [matrix aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/search-aggregations-matrix-stats-aggregation.html).

These aggregations aren't currently implemented due to the inter-plugin
communication not being set up, so they return "innerkey/matrix stats not
handled (yet)".

For matrix aggs that share a name with an existing aggregation (like 'count'),
they have be prefixed with "matrix_", so, "matrix_count", "matrix_mean", and
"matrix_variance".

Relates to elastic/x-pack-elasticsearch#2885

* Return HTTP 400 for innerkey/matrix stats aggs

* Add integration test for unimplemented matrix aggs

Original commit: elastic/x-pack-elasticsearch@34459c59aa
2017-12-15 14:33:58 -07:00
Lee Hinman 632c3e8238 SQL: Fix running `gradle run` from top-level directory (elastic/x-pack-elasticsearch#3329)
When running `gradle run` at the top level (at least with Gradle 4.4) it
attempts to run multiple instances of the server, causing the run to fail since
they can't both bind to 9200/9300.

This renames the tasks for the `qa` directories to be `runqa` and the task for
the `cli` directory to be `runcli`.

Original commit: elastic/x-pack-elasticsearch@734ab8e132
2017-12-15 08:42:18 -07:00
Nik Everett 59e6d34c29 SQL: Bundle the CLI into x-pack (elastic/x-pack-elasticsearch#3316)
This adds:

* The CLI jar itself into the `bin`. It is an executable jar.
* A shell and bat script to start the CLI. This isn't strictly required but folks will appreciate the consistency.
* Basic packaging tests for the CLI.

Relates to elastic/x-pack-elasticsearch#2979

Original commit: elastic/x-pack-elasticsearch@158f70a530
2017-12-14 09:57:03 -05:00
Nik Everett f5af60c7cf SQL: Fix error message on bad index (elastic/x-pack-elasticsearch#3312)
Fixes the error message that SQL produces when it sees unsupported
indexes. It was always returning all broken indexes as "unknown" even
though we have much better error messages. It was just throwing them
away.

I caught this originally when backporting to 6.x where we had a test
that we produced a useful error message when the user attempted to run
SQL on an index with more than one type. We couldn't run that in the
feature/sql branch because it is inside the full cluster restart tests
and the "old" version of Elasticsearch used in those tests in
feature/sql is 6.x which doesn't allow indexes with multiple types. When
I backported to 6.x the test failed because it hadn't been run before.

In addition to fixing that test and the problem, this adds another test
that will reveal the problem when run in the feature/sql and master
branch.

Original commit: elastic/x-pack-elasticsearch@c7b787baee
2017-12-14 09:56:36 -05:00
Costin Leau cdfe0f1f5a Revert "Revert "Merge branch 'feature/sql'""
This reverts commit elastic/x-pack-elasticsearch@cc79e19911.

We'll merge this when we're good and ready.

Original commit: elastic/x-pack-elasticsearch@b3ef4f2836
2017-12-13 10:19:31 -05:00
Costin Leau 2e60e831c0 Revert "Merge branch 'feature/sql'"
This reverts commit elastic/x-pack-elasticsearch@2b3f7697a5, reversing
changes made to elastic/x-pack-elasticsearch@b79f16673c.

We're backing out all of SQL.

Original commit: elastic/x-pack-elasticsearch@cc79e19911
2017-12-13 09:33:13 -05:00
Nik Everett eaa75cfdbb Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b5fc2b484d
2017-12-12 11:29:23 -05:00
Nik Everett 2ca729afc2 Fix packaging tests after breaking up x-pack
The tests were failing after breaking up x-pack. This makes them pass
locally.

Original commit: elastic/x-pack-elasticsearch@af2a7866eb
2017-12-12 11:05:51 -05:00
David Kyle 6113b86bdb [ML] Special events calendar CRUD endpoints (elastic/x-pack-elasticsearch#3267)
* Calendar CRUD endpoints

* Get calendars requires monitor permission

* Address review comments

* Add page params to get calendars

Original commit: elastic/x-pack-elasticsearch@badd1e6add
2017-12-12 09:21:44 +00:00
Nik Everett 236f64a70e SQL: Fix SSL for JDBC and CLI for real this time (elastic/x-pack-elasticsearch#3277)
Previously I'd added tests for JDBC and CLI that I *thought* used SSL but they didn't! I wasn't careful...

Testing changes:
* Actually enable SSL/HTTPS in the `qa:sql:security:ssl` subproject.
* Rework how `RemoteCli` handles security. This allows us to configure SSL, the keystore, and the username and password in a much less error prone way.
* Fix up JDBC tests to properly use SSL.
* Allow the `CliFixture` to specify the keystore location.
* Switch `CliFixture` and `RemoteCli` from sending the password in the connection string to filling out the prompt for it.
* Have `CliFixture` also send the keystore password when a keystore is configured.

This makes the following production code changes:
* Allow the CLI to configure the keystore location with the `-k`/`-keystore_location` parameters.
* If the keystore location is configured then the CLI will prompt for the password.
* Allow the configuration of urls starting with `https`.
* Improve the exception thrown when the URL doesn't parse by adding a suppressed exception with the original parse error, before we tried to add `http://` to the front of it.

Original commit: elastic/x-pack-elasticsearch@97fac4a3b4
2017-12-11 15:45:34 -05:00
Igor Motov 4bebc307c3 SQL: Add ability to close cursors (elastic/x-pack-elasticsearch#3249)
This commits adds a new end point for closing in-flight cursors, it also ensures that all cursors are properly closed by adding after test checks that ensures that we don't leave any search context open.

relates elastic/x-pack-elasticsearch#2878

Original commit: elastic/x-pack-elasticsearch@1052ea28dc
2017-12-11 11:36:02 -05:00
Igor Motov fab3712e3d SQL: Check connection on CLI startup (elastic/x-pack-elasticsearch#3278)
* SQL: Check connection on CLI startup

Adds a connection check at the CLI startup. If connection cannot be established or elasticsearch has incompatible version, the CLI doesn't start.

relates elastic/x-pack-elasticsearch#2984

Original commit: elastic/x-pack-elasticsearch@c9a58d2cd6
2017-12-11 10:00:00 -05:00
David Roberts 5fd68959a0 [ML] Make datafeeds run-as the user who created/updated them (elastic/x-pack-elasticsearch#3254)
This is the ML equivalent of what was done for Watcher in elastic/x-pack-elasticsearch#2808.

For security reasons, ML datafeeds should not run as the _xpack
user.  Instead, they record the security headers from the request
to create/update them, and reuse these when performing the search
to retrieve data for analysis.

Relates elastic/x-pack-elasticsearch#1071

Original commit: elastic/x-pack-elasticsearch@29f85de404
2017-12-11 13:01:16 +00:00
Alexander Reelsen 6bae4681e2 Tests: Replace http input integration test with REST test (elastic/x-pack-elasticsearch#3215)
Remove HttpInputIntegrationTests, which only tested existing
functionality, the remaining part was moved over into a REST tests.

relates elastic/x-pack-elasticsearch#3210

Original commit: elastic/x-pack-elasticsearch@474e5337b6
2017-12-11 13:08:58 +01:00
Igor Motov 5d0b8ff796 SQL: Rename Strings.splitStringByCommaToSet to tokenizeByCommaToSet
Relates elastic/x-pack-elasticsearch#3265

Original commit: elastic/x-pack-elasticsearch@8b06b0688b
2017-12-08 13:52:08 -05:00
Igor Motov 4330195cdf Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@f4ef2d9cdf
2017-12-08 12:35:59 -05:00
Igor Motov b72a9b5674 SQL: switch from _sql/ endpoint to _xpack/sql endpoint (elastic/x-pack-elasticsearch#3270)
As a part of xpack, SQL should use _xpack/sql endpoint instead of _sql endpoint.

relates elastic/x-pack-elasticsearch#3114

Original commit: elastic/x-pack-elasticsearch@f561b57f16
2017-12-08 12:17:26 -05:00
Alexander Reelsen eaf67f8bc2 Tests: Replace YAML tests with ESTestRestCase to be able to wait for … (elastic/x-pack-elasticsearch#3252)
* Tests: Replace YAML tests with ESTestRestCase to be able to wait for events

The YAML tests did not have any possibility to wait for the watches to
be created. A hard ten second timeout was used, that could not be
aborted, by simulating a sleep when waiting for a number of nodes that
never occured in the cluster.

This commit replaces those waiting YAML tests with ESRestTestCases, that
use `assertBusy()` to exit early once the watches have been added. Also
this increases the wait time if needed, as these tests tend to fail on
CI.

relates elastic/x-pack-elasticsearch#3217

Original commit: elastic/x-pack-elasticsearch@74b9945d88
2017-12-07 14:40:08 +01:00
Costin Leau 432f10e736 SQL: Introduce LiteralAttribute for values that are not extracted from ES (elastic/x-pack-elasticsearch#3209)
This is mainly a promotion of Literal to Attribute to better handle folding expressions from extracted queries

Original commit: elastic/x-pack-elasticsearch@c3bb48bb61
2017-12-06 19:35:34 +02:00
Igor Motov 6839f99ed0 SQL: Switch CLI to core-cli's Command and make it testable (elastic/x-pack-elasticsearch#3232)
Switches CLI to use the standard Elasticsearch Command and refactors it to be more testable. It doesn't change any cli functionality except using the bright color while displaying query results.

relates elastic/x-pack-elasticsearch#2881, elastic/x-pack-elasticsearch#3203, elastic/x-pack-elasticsearch#2990

Original commit: elastic/x-pack-elasticsearch@841f306d50
2017-12-06 11:27:51 -05:00
javanna 9664363575 remove Catalog abstraction and rename its package from catalog to index
Given that the Catalog was only ever used to hold a single index, the corresponding abstraction can be removed in favour of the abstraction that it holds, namely `GetIndexResult`.

Original commit: elastic/x-pack-elasticsearch@6932db642c
2017-12-06 13:58:17 +01:00
javanna 90aee54251 Trim down the Catalog implementations to a single one
Catalog is now a final class rather than an interface with different implementations.

relates elastic/x-pack-elasticsearch#3179

Original commit: elastic/x-pack-elasticsearch@4cc927e113
2017-12-06 13:58:17 +01:00
javanna 08950ff491 Remove security filter, replaced by get index api call which returns filtered mappings
Given that we get now filtered mappings directly from the get index API (in case security is configured with FLS), we don't need the security filter nor the filtered catalog. That means we can remove the delayed action support also from AuthorizationService and rather make SQLAction a composite action like others. It will be authorized as an action, but its indices won't be checked while that will happen with its inner actions (get index and search) which need to be properly authorized.

Also, SQLGetIndicesAction is not needed anymore, as its purpose was to retrieve the indices access resolver put in the context by the security plugin for delayed actions, which are not supported anymore.

This commit kind of reverts elastic/x-pack-elasticsearch#2162, as it is now possible to integrate with security out-of-the-box

relates elastic/x-pack-elasticsearch#2934

Original commit: elastic/x-pack-elasticsearch@64d5044426
2017-12-06 13:58:17 +01:00
javanna 4e84a1d658 [TEST] update audit logging parsing
Original commit: elastic/x-pack-elasticsearch@041cb09e2b
2017-12-05 23:00:28 +01:00
javanna 626c74a437 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@67f8321368
2017-12-05 21:50:35 +01:00
David Roberts 751caaae76 [ML] Set established model memory on job open for pre-6.1 jobs (elastic/x-pack-elasticsearch#3222)
Before this was done it was easy to get into the situation where a
job created in 5.x with a default model memory limit of 4GB could not
be opened on any node in the cluster.  Following this change this
problem will no longer occur for jobs that ran for a decent amount of
time on the old cluster.

relates elastic/x-pack-elasticsearch#3181

Original commit: elastic/x-pack-elasticsearch@cb029debba
2017-12-05 17:05:58 +00:00
Costin Leau 88b8794801 SQL: Forbid multi field groups (elastic/x-pack-elasticsearch#3199)
* SQL: GROUP BY with multiple fields are forbidden

The check is performed in the folder Verifier as the optimizer can eliminate some fields (like those with constants)

Original commit: elastic/x-pack-elasticsearch@8d49f4ab02
2017-12-05 18:41:19 +02:00
Costin Leau 4e49769efb SQL: Extend HAVING support (elastic/x-pack-elasticsearch#3155)
SQL: Extend HAVING support

Enhance Analyzer to support HAVING scalar functions over aggregates
Enhance Analyzer to push down undeclared aggs into the Aggregate
Fix bug in Analyzer$MissingRef that caused invalid groupings to still be resolved when pushed into an Aggregate
Preserve location information across the plan
Add AttributeMap as a backing for AttributeSet
Add Optimizer rule for combining projects
Add tz to DT functions toString
Change formatTemplate to not use String.format and thus to avoid
interfering with special % chars
Extend dataset with random salary and languages
Add unit tests for AttributeMap
Fix MathFunction scripting
Improve MissingRefs to enrich UnresolvedAttribute with metadata
During the Analysis unpushed attributes are automatically enriched to
provide more accurate error information
Enhance Verifier to deal with invalid (but resolved) ORDER/HAVING
Add OrderBy arithmetic tests
Improve Verifier to prevent GROUP BY on aggregations
Add tests on grouping by scalar functions

Original commit: elastic/x-pack-elasticsearch@5030d7a755
2017-12-05 18:14:15 +02:00
Dimitrios Athanasiou 6c6b72db25 [TEST] Increase timeout for CoreWithSecurityClientYamlTestSuiteIT
This suite seems to be timing out on the CI slaves. Hopefully,
some extra time will stop the failures.

Original commit: elastic/x-pack-elasticsearch@95ba86be79
2017-12-05 15:57:27 +00:00
Alexander Reelsen e3a7e4bee4 Tests: Increase timeout to wait to monitoring watches being created
Relates elastic/x-pack-elasticsearch#3217

Original commit: elastic/x-pack-elasticsearch@a6256f9e42
2017-12-05 14:20:46 +01:00
Alexander Reelsen c3e5a20242 Monitoring: Disable security for integration tests (elastic/x-pack-elasticsearch#3174)
In order to support the source directory repo split, this commit
disables security for the regular integration tests.

The MonitoringSettingsFilterTests already existed as REST test, so
this test has been removed.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@519154dd5f
2017-12-05 12:07:04 +01:00
David Roberts df9dd77656 [TEST] Fix more side effects of elastic/x-pack-elasticsearch#2975 on machines with < 16GB RAM
Our rolling upgrade tests were failing on many machines due to using
the 5.x default of 4GB model_memory_limit, which then propagated forward
to 6.1+ even though the default is now lower.

Original commit: elastic/x-pack-elasticsearch@3b23d8fe9d
2017-12-04 09:48:51 +00:00
Igor Motov 7b701cbf88 SQL: Don't override calendar for Thai locale
The Thai Buddhist calendar is close enough to Gregorian calendar that it doesn't trip H2 if timestamps with timezones are used. Related to elastic/x-pack-elasticsearch#3169.

Original commit: elastic/x-pack-elasticsearch@4f8f1b603d
2017-12-03 14:51:23 -05:00
Igor Motov a732d751c4 SQL: Replace calendar in H2 tests if a non-Gregorian calendar is detected
Instead of replacing entire locale with the ROOT locale, just strip the calendar settings if a non-Gregorian calendar is detected. Related to elastic/x-pack-elasticsearch#3169.

Original commit: elastic/x-pack-elasticsearch@33b1c63768
2017-12-03 13:33:24 -05:00
Alexander Reelsen f816b2e850 Monitoring: Move watcher tests for repository split preparations (elastic/x-pack-elasticsearch#3183)
Created a smoke-test-monitoring-with-watcher project that runs REST
tests with watcher enabled to ensure that the proper watcher are
installed either when the local or the HTTP exporter are set up.

Also removed two more watcher imports in the tests.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@0a9abc3185
2017-12-01 13:20:05 +01:00
Igor Motov d8ef52dc0d SQL: Allow H2-based tests to run only with Gregorian calendar (elastic/x-pack-elasticsearch#3170)
This PR detects non-gregorian calendars in H2 tests and overrides them. This is a temporary workaround to reduce amount of noise in CI. The permanent solution is tracked in elastic/x-pack-elasticsearch#3169.

Original commit: elastic/x-pack-elasticsearch@60991d1a11
2017-11-30 11:42:59 -05:00
Costin Leau 7cab29760d SQL: Introduce PreAnalyze phase to resolve catalogs async (elastic/x-pack-elasticsearch#2962)
SQL: Introduce PreAnalyze phase to resolve catalogs async

The new preanalyze phase collects all unresolved relations and tries
to resolve them as indices through typical async calls _before_ starting the analysis process.
The result is loaded into a catalog which is then passed to the analyzer.
While at it, the analyzer was made singleton and state across the engine
is done through SqlSession#currentContext().


Commit missing fix
Fix typo
Fix license
Fix line length
remove redundant static modifier
Remove redundant generics type
Rename catalogResolver instance member to indexResolver
Fix translate action to return a response through the listener, it hangs otherwise
IndexResolver improvements

Make sure that get index requests calls are locally executed by providing local flag.
Don't replace index/alias name with concrete index name in asCatalog response conversion. We need to preserve the original alias name for security, so it is reused in the subsequent search.
Update roles and actions names for security tests
Get index is now executed instead of sql get indices, and sql get indices has been removed.
Also made cluster privileges more restrictive to make sure that cluster state calls are no longer executed.
Fix most of the security IT tests
indices options are now unified, always lenient. The only situation where we get authorization exception back is when the user is not authorized for the sql action (besides for which indices).
Improve SessionContext handling
Fix context being invalid in non-executable phases
Make Explain & Debug command fully async
Resolve checkstyle error about redundant modifiers
Temporarily restore SqlGetIndicesAction

SqlGetIndicesAction action is still needed in RestSqlJdbcAction (metaTable and metaColumn methods), where we can't at the moment call IndexResolver directly, as security (FLS) needs index resolver to be called as part of the execution of an indices action. Once mappings are returned filtered, delayed action and the security filter will go away, as well as SqlGetIndicesAction.
SqlGetIndicesAction doesn't need to be a delayed action, my bad

[TEST] remove unused expectSqlWithAsyncLookup and rename expectSqlWithSyncLookup to expectSqlCompositeAction
Polish and feedback
Add unit test for PreAnalyzer

Original commit: elastic/x-pack-elasticsearch@57846ed613
2017-11-30 18:18:08 +02:00
Igor Motov 6fceb2fdde SQL: Fix more JDK vs Joda TZ issues (elastic/x-pack-elasticsearch#3137)
Adds a random timezone known to both JDK and Joda as a connection parameter to all JdbcIntegrationTestCase-based tests.

Original commit: elastic/x-pack-elasticsearch@6be6a3b69b
2017-11-29 19:15:29 -05:00
Igor Motov d5525f38f3 SQL: Return proper NUM_PREC_RADIX for non-numeric types (elastic/x-pack-elasticsearch#3086)
NUM_PREC_RADIX should be 10 for integer types, 2 for rational types and null for everything else.

relates elastic/x-pack-elasticsearch#3085

Original commit: elastic/x-pack-elasticsearch@81d5ee04b3
2017-11-29 08:21:42 -05:00
David Kyle 171c48fd2f [TESTS] Refactor yml test suite classes (elastic/x-pack-elasticsearch#3145)
* Use XPackRestIT as base class for XDocsClientYamlTestSuiteIT

* Remove the XPackRestTestCase class

* Address review comments

* Fix checkstyle checks

Original commit: elastic/x-pack-elasticsearch@c2a5e60c12
2017-11-29 12:43:53 +00:00
Nik Everett 0cc153f6d3 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@ccc2fc708e
2017-11-28 13:24:36 -05:00
Igor Motov da180bd9f9 Tests: Muted FullClusterRestartIT.testMonitoring
Tracked by elastic/x-pack-elasticsearch#3068

Original commit: elastic/x-pack-elasticsearch@199d4bb6b9
2017-11-28 12:06:10 -05:00
Nik Everett 18e88122eb SQL: Add more error integration tests (elastic/x-pack-elasticsearch#3134)
We didn't have many integration tests for errors other than the security
errors. This adds some and sets up a way we can make sure we are
consistent across the REST, JDBC, and CLI.

relates elastic/x-pack-elasticsearch#3033

Original commit: elastic/x-pack-elasticsearch@debbb2ec46
2017-11-28 11:54:51 -05:00
Alexander Reelsen cdb85d8317 Watcher: Run tests without security enabled (elastic/x-pack-elasticsearch#3060)
In order to support the repository split, this changes the
`AbstractWatcherIntegrationTestCase` to not run with security enabled.

We have a dedicated QA project called `smoke-test-watcher-with-security`,
where tests that explicitely need security should be running.

This commit removes the possibility to enable security as part of the
test case. In addition some tests have been moved over to the dedicated
project.

In addition the `timewarp` functionality cannot be configured with a
system property anymore. This would not have worked anyway, because
tests were already dependent on that  functionality and did not have any
other means of running. A bit of redundant code was removed due to this.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@b24b365ad1
2017-11-28 13:11:49 +01:00
Nik Everett df802b40c8 SQL: Rework SSL testing (elastic/x-pack-elasticsearch#3126)
Drop the ssl tests against the java builtin https server. They were
failing and the failures were undebuggable. I still don't know what was
happening because you can't get any logging out of the server.

Add SSL tests against Elasticsearch because that is what actually needs
to work.

relates elastic/x-pack-elasticsearch#2870

Original commit: elastic/x-pack-elasticsearch@284cf7fb58
2017-11-27 18:52:16 -05:00
Igor Motov 5c88fa0b3b SQL: Add support for plain text output to /_sql endpoint (elastic/x-pack-elasticsearch#3124)
The /_sql endpoint now returns the results in the text format by default. Structured formats are also supported using the format parameter or accept header similar to _cat endpoints.

Original commit: elastic/x-pack-elasticsearch@4353793b83
2017-11-27 18:10:13 -05:00
Igor Motov 626e9b87a1 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@6c5a683209
2017-11-27 09:26:14 -05:00