* SQL: Move shared REST client methods to shared-client
This commit is a preliminary step for moving JDBC to the REST client. It extracts the common REST clients from CLI and moves it to shared-client. This will allow us to move to the 5 project setup: rest-proto, shared-client, server, jdbc, cli with the following dependencies:
server <-- rest-proto
shared-client <-- rest-proto
jdbc <-- shared-client
cli <-- shared-client
Relates to elastic/x-pack-elasticsearch#3419
Original commit: elastic/x-pack-elasticsearch@2e6a134de0
* SQL: Add support for object/inner/dotted fields
Improve validation when selecting compound fields
Improve fallback to exact field for text fields
Consolidate various field attributes into one
Consolidate various field HitExtractors into one
Improve grammar for column declaration
Update response of embedded server
Improve index validation to allow use of internal indices
Add clean-up of scroll in case of server errors
By default thrown an exception on multi-valued fields
Original commit: elastic/x-pack-elasticsearch@45b73fe0dc
* Starts to build the list of supported functions, adding links to
wikipedia when there is any doubt what the functions mean.
* Extracts an example of using the function from the test suite.
* Explicitly calls out how we round (half up) because there are
lots of ways to round.
Original commit: elastic/x-pack-elasticsearch@5fb64ba869
This adds support for (almost) all of the options that the `query_string` query
supports.
Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `QUERY()` query.
Relates to elastic/x-pack-elasticsearch#3361
Original commit: elastic/x-pack-elasticsearch@da8b29b53c
* SQL: Add all MATCH() query options
This adds support for (almost) all of the options that the `match` query
supports.
Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `MATCH()` query.
Relates to elastic/x-pack-elasticsearch#3361
* Add getters required for Node usage
Original commit: elastic/x-pack-elasticsearch@144e2bec02
We don't support the SQL `EXISTS` keyword. It looks like:
```
SELECT * FROM test WHERE EXISTS (SELECT * FROM foo WHERE test.id =
foo.id)
```
It is basically a `JOIN` that doesn't return columns. Since we don't
support `JOIN`, we don't support `EXISTS`.
This PR improves the error message from "unresolved blah blah blah" to
"EXISTS is not yet supported".
relates elastic/x-pack-elasticsearch#3176
Original commit: elastic/x-pack-elasticsearch@548d57c8c1
`SELECT DISTINCT foo FROM bar` is not yet implemented and was returning
an "unplanned item" error which isn't useful to users so I replaced it
with `SELECT DISTINCT is not yet supported`.
`SELECT foo, COUNT(*) FROM bar GROUP BY ALL foo` is not supported.
Specifically the `ALL` part. It is fairly esoteric so see [1] for what
it does. Regardless of what it does it is not widely supported and even
Microsoft's SQL Server has deprecated it so we should never support it.
Probably.
[1]: https://technet.microsoft.com/en-us/library/ms175028(v=sql.90).aspx
Related to elastic/x-pack-elasticsearch#3176
Original commit: elastic/x-pack-elasticsearch@56e5ca3009
`JOIN`s aren't supported right yet so we should send back a nice 400
level error to the user telling them that.
Also pulls out some common code in `RestSqlTestCase` that I've been
staring at for a while.
Relates to elastic/x-pack-elasticsearch#3176
Original commit: elastic/x-pack-elasticsearch@1c1bd1c355
* Add remaining matrix aggregations
This adds the remaining [matrix aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/search-aggregations-matrix-stats-aggregation.html).
These aggregations aren't currently implemented due to the inter-plugin
communication not being set up, so they return "innerkey/matrix stats not
handled (yet)".
For matrix aggs that share a name with an existing aggregation (like 'count'),
they have be prefixed with "matrix_", so, "matrix_count", "matrix_mean", and
"matrix_variance".
Relates to elastic/x-pack-elasticsearch#2885
* Return HTTP 400 for innerkey/matrix stats aggs
* Add integration test for unimplemented matrix aggs
Original commit: elastic/x-pack-elasticsearch@34459c59aa
When running `gradle run` at the top level (at least with Gradle 4.4) it
attempts to run multiple instances of the server, causing the run to fail since
they can't both bind to 9200/9300.
This renames the tasks for the `qa` directories to be `runqa` and the task for
the `cli` directory to be `runcli`.
Original commit: elastic/x-pack-elasticsearch@734ab8e132
Fixes the error message that SQL produces when it sees unsupported
indexes. It was always returning all broken indexes as "unknown" even
though we have much better error messages. It was just throwing them
away.
I caught this originally when backporting to 6.x where we had a test
that we produced a useful error message when the user attempted to run
SQL on an index with more than one type. We couldn't run that in the
feature/sql branch because it is inside the full cluster restart tests
and the "old" version of Elasticsearch used in those tests in
feature/sql is 6.x which doesn't allow indexes with multiple types. When
I backported to 6.x the test failed because it hadn't been run before.
In addition to fixing that test and the problem, this adds another test
that will reveal the problem when run in the feature/sql and master
branch.
Original commit: elastic/x-pack-elasticsearch@c7b787baee
Previously I'd added tests for JDBC and CLI that I *thought* used SSL but they didn't! I wasn't careful...
Testing changes:
* Actually enable SSL/HTTPS in the `qa:sql:security:ssl` subproject.
* Rework how `RemoteCli` handles security. This allows us to configure SSL, the keystore, and the username and password in a much less error prone way.
* Fix up JDBC tests to properly use SSL.
* Allow the `CliFixture` to specify the keystore location.
* Switch `CliFixture` and `RemoteCli` from sending the password in the connection string to filling out the prompt for it.
* Have `CliFixture` also send the keystore password when a keystore is configured.
This makes the following production code changes:
* Allow the CLI to configure the keystore location with the `-k`/`-keystore_location` parameters.
* If the keystore location is configured then the CLI will prompt for the password.
* Allow the configuration of urls starting with `https`.
* Improve the exception thrown when the URL doesn't parse by adding a suppressed exception with the original parse error, before we tried to add `http://` to the front of it.
Original commit: elastic/x-pack-elasticsearch@97fac4a3b4
This commits adds a new end point for closing in-flight cursors, it also ensures that all cursors are properly closed by adding after test checks that ensures that we don't leave any search context open.
relates elastic/x-pack-elasticsearch#2878
Original commit: elastic/x-pack-elasticsearch@1052ea28dc
This is mainly a promotion of Literal to Attribute to better handle folding expressions from extracted queries
Original commit: elastic/x-pack-elasticsearch@c3bb48bb61
Given that the Catalog was only ever used to hold a single index, the corresponding abstraction can be removed in favour of the abstraction that it holds, namely `GetIndexResult`.
Original commit: elastic/x-pack-elasticsearch@6932db642c
Given that we get now filtered mappings directly from the get index API (in case security is configured with FLS), we don't need the security filter nor the filtered catalog. That means we can remove the delayed action support also from AuthorizationService and rather make SQLAction a composite action like others. It will be authorized as an action, but its indices won't be checked while that will happen with its inner actions (get index and search) which need to be properly authorized.
Also, SQLGetIndicesAction is not needed anymore, as its purpose was to retrieve the indices access resolver put in the context by the security plugin for delayed actions, which are not supported anymore.
This commit kind of reverts elastic/x-pack-elasticsearch#2162, as it is now possible to integrate with security out-of-the-box
relates elastic/x-pack-elasticsearch#2934
Original commit: elastic/x-pack-elasticsearch@64d5044426
* SQL: GROUP BY with multiple fields are forbidden
The check is performed in the folder Verifier as the optimizer can eliminate some fields (like those with constants)
Original commit: elastic/x-pack-elasticsearch@8d49f4ab02
SQL: Extend HAVING support
Enhance Analyzer to support HAVING scalar functions over aggregates
Enhance Analyzer to push down undeclared aggs into the Aggregate
Fix bug in Analyzer$MissingRef that caused invalid groupings to still be resolved when pushed into an Aggregate
Preserve location information across the plan
Add AttributeMap as a backing for AttributeSet
Add Optimizer rule for combining projects
Add tz to DT functions toString
Change formatTemplate to not use String.format and thus to avoid
interfering with special % chars
Extend dataset with random salary and languages
Add unit tests for AttributeMap
Fix MathFunction scripting
Improve MissingRefs to enrich UnresolvedAttribute with metadata
During the Analysis unpushed attributes are automatically enriched to
provide more accurate error information
Enhance Verifier to deal with invalid (but resolved) ORDER/HAVING
Add OrderBy arithmetic tests
Improve Verifier to prevent GROUP BY on aggregations
Add tests on grouping by scalar functions
Original commit: elastic/x-pack-elasticsearch@5030d7a755
SQL: Introduce PreAnalyze phase to resolve catalogs async
The new preanalyze phase collects all unresolved relations and tries
to resolve them as indices through typical async calls _before_ starting the analysis process.
The result is loaded into a catalog which is then passed to the analyzer.
While at it, the analyzer was made singleton and state across the engine
is done through SqlSession#currentContext().
Commit missing fix
Fix typo
Fix license
Fix line length
remove redundant static modifier
Remove redundant generics type
Rename catalogResolver instance member to indexResolver
Fix translate action to return a response through the listener, it hangs otherwise
IndexResolver improvements
Make sure that get index requests calls are locally executed by providing local flag.
Don't replace index/alias name with concrete index name in asCatalog response conversion. We need to preserve the original alias name for security, so it is reused in the subsequent search.
Update roles and actions names for security tests
Get index is now executed instead of sql get indices, and sql get indices has been removed.
Also made cluster privileges more restrictive to make sure that cluster state calls are no longer executed.
Fix most of the security IT tests
indices options are now unified, always lenient. The only situation where we get authorization exception back is when the user is not authorized for the sql action (besides for which indices).
Improve SessionContext handling
Fix context being invalid in non-executable phases
Make Explain & Debug command fully async
Resolve checkstyle error about redundant modifiers
Temporarily restore SqlGetIndicesAction
SqlGetIndicesAction action is still needed in RestSqlJdbcAction (metaTable and metaColumn methods), where we can't at the moment call IndexResolver directly, as security (FLS) needs index resolver to be called as part of the execution of an indices action. Once mappings are returned filtered, delayed action and the security filter will go away, as well as SqlGetIndicesAction.
SqlGetIndicesAction doesn't need to be a delayed action, my bad
[TEST] remove unused expectSqlWithAsyncLookup and rename expectSqlWithSyncLookup to expectSqlCompositeAction
Polish and feedback
Add unit test for PreAnalyzer
Original commit: elastic/x-pack-elasticsearch@57846ed613
Adds a random timezone known to both JDK and Joda as a connection parameter to all JdbcIntegrationTestCase-based tests.
Original commit: elastic/x-pack-elasticsearch@6be6a3b69b
Drop the ssl tests against the java builtin https server. They were
failing and the failures were undebuggable. I still don't know what was
happening because you can't get any logging out of the server.
Add SSL tests against Elasticsearch because that is what actually needs
to work.
relates elastic/x-pack-elasticsearch#2870
Original commit: elastic/x-pack-elasticsearch@284cf7fb58
The /_sql endpoint now returns the results in the text format by default. Structured formats are also supported using the format parameter or accept header similar to _cat endpoints.
Original commit: elastic/x-pack-elasticsearch@4353793b83
Adds the option to specify an elasticsearch filter in addition to the SQL query by introducing a filter parameter in the REST query which would create a boolean filter if the SQL query generates an elasticsearch query or a constant score query if SQL if the SQL query doesn't generates an elasticsearch query. Usage:
{
"query": "SELECT * FROM index",
"filter" : { "term" : { "tag" : "tech" } }
}
relates elastic/x-pack-elasticsearch#2895
Original commit: elastic/x-pack-elasticsearch@9a73813c7f
The IDEs don't participate in the shading but gradle does. So we have to
be a little more tricky about how we set up the IDE projects, sadly.
Original commit: elastic/x-pack-elasticsearch@5196756702
This teaches SQL to parse Elasticsearch's standard error responses
but doesn't change SQL to general Elasticsearch's standard error responses
in all cases. That can come in a followup. We do this parsing with
jackson-core, the same dependency Elasticsearch uses for parsing
json. We shade jackson-core in the JDBC driver so that users don't have to worry about
dependency clashes. We do not do so in the CLI because it is a standalone
application.
We get a few "bonus" changes along the way:
1. We save a copy operation. Before this change responses were spooled
into memory and then parsed. After this change they are parsed directly
from the response stream.
2. We had a few classes entirely to support the spooling operation that we
no longer need: `BytesArray`, `FastByteArrayInputStream`, and
`BasicByteArrayOutputStream`.
3. SQL's `Version` was incorrectly parsing the version from the jar manifest.
We didn't notice because the test was rigged to return `UNKNOWN` because
we *were* running the test from the compiled classes directory instead of the
jar. As part of shading jackson we moved running the tests to running against
the shaded jar. Now we can actually assert that we parse the version correctly.
It turns out we weren't. So I fixed it.
Original commit: elastic/x-pack-elasticsearch@2e8f397bf4
1. decouple JdbcDriver from other classes to not trigger static
initialization (this happens through JDBC service discovery)
2. reduce visibility of JdbcDriver#close so only on jar unloading it
gets triggered
3. mark 3 methods introduced in Jdbc 4.1 as unsupported (their semantics
are somewhat weird)
4. Move versioning info in one class
5. Hook Version class in both JDBC entry points to perform cp sanity
checks
6. Remove JdbcDataSource#close (DebugLog are closed when the Driver gets
unloaded by the DriverManager) as there can be multiple instances of DS
but only one for Driver known by the DriverManager
Replace Strings with constants
Properly set TZ in security tests as well
JdbcDataSource is more defensive with its internal properties
JdbcConfiguration password parameter is aligned with JDBC DriverManager
Remove usage of JdbcConnection API
Removed JdbcConnection#setTimeZone - this encourages folks to use our
private API which would tie us down.
It is somewhat limiting for folks but it has less downsides overall and
does not trip debugging (which adds a proxy unaware of this method).
Update docs
Add JdbcDataSource into the Jdbc suite
Original commit: elastic/x-pack-elasticsearch@c713665d53
* Fix several NOCOMMITS
- renamed Assert to Check to make the intent clear
- clarify esMajor/Minor inside connection (thse are actually our own
methods, not part of JDBC API)
- wire pageTimeout into Cursor#nextPage
Original commit: elastic/x-pack-elasticsearch@7626c0a44a
JodaTime timezone db can be out of date compared to that of the JDK which causes the JDBC Connection to fail when the randomized tests pick a timezone that's available in the JDK but not in Joda, like SystemV/PST8. This is happening because JdbcConnection configuration is using system default timezone and tries to pass it to Elasticsearch that is using joda. This commit, explicitly sets the time zone on JdbcConnection to a time zone randomly selected from a list of timezones that are known to both JDK and Joda.
relates elastic/x-pack-elasticsearch#2812
Original commit: elastic/x-pack-elasticsearch@b02e9794a8
If you wrote a test at the end of one of SQL's test spec files that was
just a name without a body then the parser would throw the test away.
Now it fails to intiaize the class with an error message telling you
which file is broken.
Original commit: elastic/x-pack-elasticsearch@023a942ca3
Fix the name of the action the SQL uses to lookup index information from
the cluster state. The old name was silly.
Original commit: elastic/x-pack-elasticsearch@805fb29662
I realized that we weren't running our DatabaseMetaData tests. One thing led to another and I made these changes:
1. Got the DatabaseMetaData tests running in all three of our QA projects.
2. Fixed the SecurityCatalogFilter to work with `SqlGetIndicesAction`. It worked before, but only for requests that were a `SqlAction` as well as `SqlGetIndicesAction`.
3. Added security test for the JDBC DatabaseMetaData requests. These mirror exactly the security tests that we use for `SHOW TABLES` and `DESCRIBE` but cover the JDBC actions.
Original commit: elastic/x-pack-elasticsearch@7026d83c06
Adds docs for the REST API, translate API, the CLI, and JDBC.
Next we need to add more example queries and documentation for our
extensions.
Original commit: elastic/x-pack-elasticsearch@ed6d1360d2
* Remove usage of Settings inside SqlSettings
Also hook client timeouts to the backend
Set UTC as default timezone when using CSV
As the JVM timezone changes, make sure to pin it to UTC since this is what the results are computed against
Original commit: elastic/x-pack-elasticsearch@3e7aad8c1f
This adds all of the security tests I think SQL is going to need for the initial release. SQL is still missing an entire scenario though: SSL enabled. Either way, this removes some `NOCOMMIT`s in `qa/sql/security`. Adding the SSL testing can come later.
Original commit: elastic/x-pack-elasticsearch@851620b606
Add security tests for SQL's CLI and JDBC features. I do this by factoring out all the "actions" from the existing REST tests into an interface and implement it for REST, CLI, and JDBC. This way we can share the same audit log assertions across tests and we can be sure that the REST, CLI, and JDBC tests cover all the same use cases.
Original commit: elastic/x-pack-elasticsearch@82ff66a520
This is *way* faster because we don't have to wait for the audit
events from previous test runs to drain into the index. And we
don't have to wait for the index's refresh cycle. We have to parse
the log lines which is a bit more brittle but it feels worth it
at this point.
Original commit: elastic/x-pack-elasticsearch@4b1758fc32
Firstly, data in H2 is now stored in TIMESTAMP WITH TIME ZONE since H2
does not allow a global TZ to be set and picks the JVM TZ when a record
is read.
JdbcAssert is now aware of this allows TIMESTAMP with TZ == TIMESTAMP
Discovered a serious bug in DateTimeFunction - unfortunately date
histogram is not useful except for year since most extract functions
avoid ordering which a histogram preserves.
Thus most DTF are now terms aggs with scripting.
Improved a bug that caused duplicate functions to not be detected because
of aliasing.
Moved some datetime tests to CSV but the aggs tests now are in sync with
H2
Fixed bug that caused arithmetic on aggs to not be properly resolved by
splitting the processor definition tree to aggName (unresolved) and
aggPath (resolved)
Original commit: elastic/x-pack-elasticsearch@e75ada68f1
It was disabled because the CLI didn't work with security but
we've since fixed that so we can enable it.
Original commit: elastic/x-pack-elasticsearch@8d9b5ad89b
We weren't returning errors correctly from the server
or catching them correctly in the CLI. This fixes that
and adds simple integration tests.
Original commit: elastic/x-pack-elasticsearch@259da0da6f
* Improve JDBC communication
Jdbc HTTP client uses only one url for messages and relies on / for ping
Fixed ES prefix being discarded (missing /)
Add HEAD handler for JDBC endpoint
Original commit: elastic/x-pack-elasticsearch@389f82262e
We put the CLI in unix mode so if we send it
`\r\n` (the default in windows) then it'll
spit out extra "you are on a line continuation"
characters (`|`). Instead, we can use `\n`
directly and everything works.
I've also added a timeout to the reads from the
CLI because it makes the tests easier to debug.
Original commit: elastic/x-pack-elasticsearch@69f69f4092
SQL was cleaning up the audit logs *after* each test
but this switches it to cleaning up the audit logs
*before* each test. This is faster because we can
generate a lot of audit logs before the first test.
Original commit: elastic/x-pack-elasticsearch@71d8f76667
Now that we have fetch size working consistently we should randomize
the fetch size that we use in the tests to detect any errors caused
by strange fetch sizes.
Original commit: elastic/x-pack-elasticsearch@2c41fb5309
* Switch `ResultSet#getFetchSize` from returning the *requested*
fetch size to returning the size of the current page of results.
For scrolling searches without parent/child this'll match the
requested fetch size but for other things it won't. The nice thing
about this is that it lets us tell users a thing to look at if
they are wondering why they are using a bunch of memory.
* Remove all the entire JDBC action and implement it on the REST
layer instead.
* Create some code that can be shared between the cli and jdbc
actions to properly handle errors and request deserialization so
there isn't as much copy and paste between the two. This helps
because it calls out explicitly the places where they are different.
* I have not moved the CLI REST handle to shared code because
I think it'd be clearer to make that change in a followup PR.
* There is now no more need for constructs that allow iterating
all of the results in the same request. I have not removed these
because I feel that that cleanup is big enough to deserve its own
PR.
Original commit: elastic/x-pack-elasticsearch@3b12afd11c
Column type autodetect of integer types is broken in JDBC CSV library when it is used in tr-TR locale. The library is using toLowerCase() calls with default locale, which causes it to convert autodetected type name "Int" to lowercase "ınt" in tr-TR locale and not recognize it as an int type afterwards.
This commit adds a temporary workaround that makes the prevents that test from failing by specifying explicit column types in all tests where integer columns are present.
Original commit: elastic/x-pack-elasticsearch@86ca2acd8c
Builds on elastic/x-pack-elasticsearch#2403 to move all of sql's integration testing into
qa modules with different running server configurations. The
big advantage of this is that it allows us to test the cli and
jdbc with security present.
Creating a project that depends on both cli and jdbc and the
server has some prickly jar hell issues because cli and jdbc
package their dependencies in the jar. This works around it
in a few days:
1. Include only a single copy of the JDBC dependencies with
careful gradle work.
2. Do not include the CLI on the classpath at all and instead
run it externally.
I say "run it externally" rather than "fork it" because Elasticsearch
tests aren't allowed to fork other processes. This is forbidden
by seccomp on linux and seatbelt on osx and cannot be explicitly
requested like additional security manager settings. So instead
of forking the CLI process directly the tests interact with a test
fixture that isn't bound by Elasticsearch's rules and *can* fork
it.
This forking of the CLI has a nice side effect: it forces us to
make sure that things like security and connection strings other
than `localhost:9200` work. The old test could and did work around
missing features like that. The new tests cannot so I added the
ability to set the connection string. Configuring usernames and
passwords was also not supported but I did not add support for
that, only created the failing test and marked it as `@AwaitsFix`.
Original commit: elastic/x-pack-elasticsearch@560c6815e3
Instead of throwing and catching an exception for invalid
indices this returns *why* they are invalid in a convenient
object form that can be thrown as an exception when the index
is required or the entire index can be ignored when listing
indices.
Original commit: elastic/x-pack-elasticsearch@f45cbce647
This integrates SQL's metadata calls with security by creating
`SqlIndicesAction` and routing all of SQL's metadata calls through
it. Since it *does* know up from which indices it is working against
it can be an `IndicesRequest.Replaceable` and integrate with the
existing security infrastructure for filtering indices.
This request is implemented fairly similarly to the `GetIndexAction`
with the option to read from the master or from a local copy of
cluster state. Currently SQL forces it to run on the local copy
because the request doesn't properly support serialization. I'd
like to implement that in a followup.
Original commit: elastic/x-pack-elasticsearch@15f9512820
This shuffles all of SQL's QA tests into the `qa/sql` directory, moving
some shared resources into the new `qa:sql` project. It also rigs up
testing of the rest SQL interface in all the sql qa configurations:
without security, with security, and against multiple nodes.
I've had to make some modifications to how we handle the audit log
because it has gotten pretty slow. If these modifications turn out to
not be fast enough then I'll change the test to querying the log files
and drop the audit log index entirely but the index seems to be holding
out for now.
Original commit: elastic/x-pack-elasticsearch@ff3b5a74c1
Igor Motov
Costin Leau
Costin Leau
Nik Everett
Lee Hinman
javanna