Remove functions without a backing matrix agg
MatrixAgg works across multiple fields and exposing it directly in SQL
does not work. Instead isolated functions are exposed which get folded
and optimized into one matrix agg per field. Thus not all matrix
functions can be exposed in SQL, at least at this time.
Instead of depending on the plugin directly, depend on the plugin client
jar (matrix-agg-client)
Remove outdated test
Original commit: elastic/x-pack-elasticsearch@ec9b31bf59
The type of BUFFER_LENGTH needs to be an integer (not NULL) and unsigned
indicate the opposite of signed.
Change isSigned from Object to primitive
Since all the consumer of isSigned expect a primitive, an Object is just causing troubles by being null.
Update description table
Original commit: elastic/x-pack-elasticsearch@8e1960bdb5
I'm really really sad to be removing the cli-fixture but I've had
trouble with it leaking recently it is pretty slow. Beyond that, we'd
prefer that our test fixture only fixture things that are external
depndencies.
So, yeah, I'm removing it. So we get faster tests and no chance of
leaking processes. We lose some "realness" in the tests. Instead of
interacting with the CLI like a real user we embed it in the test
process. That means we don't test the forking, we don't test the
executable jar, and we don't test the jLine console detection stuff. On
the other hand we were kind of forcing the jLine console detection stuff
in a funky way with the fixture anyway. And we test the executable jar
in the packaging tests. And that'll have to do.
I haven't renamed `RemoteCli` because it'd bloat this commit with
mechanical changes that'd make it hard to review. I'll rename it in a
followup commit.
This also updates jLine so we can disable blinking to matching
parentheses during testing. I have no clue why, but this wasn't
happening when we used the fixture. The trouble with the blinking is
that it is based on *time* so it slows things down. Worse, it works
inconsistently! Sometimes it spits out sensible ascii codes and
sometimes it, well, spits out weird garbage. When you use it in person
it works fine though. So we keep it on when not testing.
Cleans up some redundancy in when testing CLI errors. Less copy and
paste good.
I was tempted to disable the xterm emulation entirely while working on
this because upgrading jLine changed a few things and it was a real pain
to update. But If we turned that off then we'd have *nothing* testing
the colors and such. That'd be a shame because we use color in the
output to commicate stuff. I like it so I don't want to break it.
While I was there, I replaces the cli connector's `PrintWriter` with a
`BufferedWriter`. The `PrintWriter` was kind of a trap because `println`
would fail to work properly on windows because we force the terminal
into xterm mode and it doesn't know what to do with windows line
endings. Windows.....
Additionally I fixed a race condition between disabling echo when
reading passwords and fast writers. We were disabling the echo shortly
after sending the prompt. A fast enough writer could send us text before
the echo disable kicked in. Now I delegate to `LineReader#readLine`
with a special echo mask that disables echo. This is both easier to test
and doesn't seem to have the race condition. This race condition was
failing the tests because they are so much faster now. Yay!
Original commit: elastic/x-pack-elasticsearch@d0ec027396
Add basic support for catalog parameters in SYS COLUMN
Pass an empty string instead of a null inside the prepared statement
Don't use pattern for catalog in getColumns
Original commit: elastic/x-pack-elasticsearch@17e9e851a0
Introduce system commands as alternative to meta HTTP endpoints
Pass in cluster name
Use 'BASE TABLE' instead of 'INDEX' when describing a table to stick
with the SQL terminology
Original commit: elastic/x-pack-elasticsearch@600312b8f7
Adds documentation for all of the date time functions using the new
cli-like format extracted from the csv spec. In the process of doing
this I noticed that the `WEEK` function isn't exposed as a function.
This exposes it for consistency.
Relates to elastic/x-pack-elasticsearch#2898
Original commit: elastic/x-pack-elasticsearch@0459b24cb9
This allows CSV tests to include a line between the header and the
values that is ignored by the test framework. This optional line can be
added to the tests to make them a little easier to read which is useful
when they are included in the documentation. As a side effect they also
closely mimick the output of the CLI. To the point where you can copy
directly from the CLI and paste into the CSV tests.
Example:
```
constantYear
// tag::year
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
year
---------------
2018
// end::year
;
```
This can be extracted with a construct like this in the docs:
```
["source","sql",subs="attributes,callouts,macros"]
--------------------------------------------------
include-tagged::{sql-specs}/datetime.csv-spec[year]
--------------------------------------------------
```
Which makes documentation that looks like this:
```
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
year
---------------
2018
```
Which is fairly nice.
Original commit: elastic/x-pack-elasticsearch@8c10b5cb10
I went to write some docs for datetime functions that look like:
```
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
year
2018
```
because I figured they'd be pretty easy to read because they didn't
require any knowledge of a data set. But it turns out that constant
folding doesn't work properly for date time functions because they don't
actually apply the extraction.
Original commit: elastic/x-pack-elasticsearch@aa9c66b2c7
The on-closing Connection is closed so there's no leak however closing
the ResultSet is good practice and tests another piece of code as well
Original commit: elastic/x-pack-elasticsearch@3a9cee70a3
* SQL: Wrap coercion in JdbcResultSet to throw SQLException
This catches the `ClassCastException` that could be thrown when retrieving data
from a result set, instead converting it into a `SQLException`.
Resolveselastic/x-pack-elasticsearch#3207
* Add simple test for incorrect coercion
Original commit: elastic/x-pack-elasticsearch@5480a48d95
* SQL: update support for nested docs
Expand sample data with nested docs
Optimize source generation for nested parents
Nested objects fields are specified in inner-hits and don't need to be
specified at the top-level query. Further more disable parent source if
not needed.
ComputingFieldRef needs to return the hitName of its wrapped children
otherwise nested values cannot be extracted
Disable GROUP BY/HAVING on nested fields
Update FieldAttribute requirements in the random tests
Original commit: elastic/x-pack-elasticsearch@e44951b5f6
Adds a mode parameter to all SQL-related requests. The mode parameter is used for license checks as well as to define the response content. For now only two modes are supported plain (default) and jdbc. We will add other modes in the future as we add more clients.
Relates elastic/x-pack-elasticsearch#3419
Original commit: elastic/x-pack-elasticsearch@b49ca38d4b
Previously multi-line CLI SQL statements were joined, but the space command was
missing, so a command like:
```
sql> SHOW
| functions;
```
Would incorrectly parse as "showfunctions" and throw an error.
This fixes the behavior and adds a test for multi-line commands.
Resolveselastic/x-pack-elasticsearch#3410
Original commit: elastic/x-pack-elasticsearch@3870924ccd
Tweak the grammar to differentiate between table and normal identifier.
The table one allows wildcard while the normal one (for fields) does not.
Original commit: elastic/x-pack-elasticsearch@a714e950db
Add support for aliases and indices pattern
Enhance ShowTable info to differentiate between aliases and indices
Add regex filtering of index names
Handle security exceptions (in case of no privileges or no matching)
Original commit: elastic/x-pack-elasticsearch@91e3674ca7
We test on Windows with the terminal in unix mode, so we need to make sure we send `\n` instead of `\r\n`. In this case the extra `\n` after password was causing an extra new line in tests breaking them on Windows.
Original commit: elastic/x-pack-elasticsearch@721ba32bd6
* SQL: Move shared REST client methods to shared-client
This commit is a preliminary step for moving JDBC to the REST client. It extracts the common REST clients from CLI and moves it to shared-client. This will allow us to move to the 5 project setup: rest-proto, shared-client, server, jdbc, cli with the following dependencies:
server <-- rest-proto
shared-client <-- rest-proto
jdbc <-- shared-client
cli <-- shared-client
Relates to elastic/x-pack-elasticsearch#3419
Original commit: elastic/x-pack-elasticsearch@2e6a134de0
* SQL: Add support for object/inner/dotted fields
Improve validation when selecting compound fields
Improve fallback to exact field for text fields
Consolidate various field attributes into one
Consolidate various field HitExtractors into one
Improve grammar for column declaration
Update response of embedded server
Improve index validation to allow use of internal indices
Add clean-up of scroll in case of server errors
By default thrown an exception on multi-valued fields
Original commit: elastic/x-pack-elasticsearch@45b73fe0dc
* Starts to build the list of supported functions, adding links to
wikipedia when there is any doubt what the functions mean.
* Extracts an example of using the function from the test suite.
* Explicitly calls out how we round (half up) because there are
lots of ways to round.
Original commit: elastic/x-pack-elasticsearch@5fb64ba869
This adds support for (almost) all of the options that the `query_string` query
supports.
Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `QUERY()` query.
Relates to elastic/x-pack-elasticsearch#3361
Original commit: elastic/x-pack-elasticsearch@da8b29b53c
* SQL: Add all MATCH() query options
This adds support for (almost) all of the options that the `match` query
supports.
Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `MATCH()` query.
Relates to elastic/x-pack-elasticsearch#3361
* Add getters required for Node usage
Original commit: elastic/x-pack-elasticsearch@144e2bec02
We don't support the SQL `EXISTS` keyword. It looks like:
```
SELECT * FROM test WHERE EXISTS (SELECT * FROM foo WHERE test.id =
foo.id)
```
It is basically a `JOIN` that doesn't return columns. Since we don't
support `JOIN`, we don't support `EXISTS`.
This PR improves the error message from "unresolved blah blah blah" to
"EXISTS is not yet supported".
relates elastic/x-pack-elasticsearch#3176
Original commit: elastic/x-pack-elasticsearch@548d57c8c1
`SELECT DISTINCT foo FROM bar` is not yet implemented and was returning
an "unplanned item" error which isn't useful to users so I replaced it
with `SELECT DISTINCT is not yet supported`.
`SELECT foo, COUNT(*) FROM bar GROUP BY ALL foo` is not supported.
Specifically the `ALL` part. It is fairly esoteric so see [1] for what
it does. Regardless of what it does it is not widely supported and even
Microsoft's SQL Server has deprecated it so we should never support it.
Probably.
[1]: https://technet.microsoft.com/en-us/library/ms175028(v=sql.90).aspx
Related to elastic/x-pack-elasticsearch#3176
Original commit: elastic/x-pack-elasticsearch@56e5ca3009
`JOIN`s aren't supported right yet so we should send back a nice 400
level error to the user telling them that.
Also pulls out some common code in `RestSqlTestCase` that I've been
staring at for a while.
Relates to elastic/x-pack-elasticsearch#3176
Original commit: elastic/x-pack-elasticsearch@1c1bd1c355
* Add remaining matrix aggregations
This adds the remaining [matrix aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/search-aggregations-matrix-stats-aggregation.html).
These aggregations aren't currently implemented due to the inter-plugin
communication not being set up, so they return "innerkey/matrix stats not
handled (yet)".
For matrix aggs that share a name with an existing aggregation (like 'count'),
they have be prefixed with "matrix_", so, "matrix_count", "matrix_mean", and
"matrix_variance".
Relates to elastic/x-pack-elasticsearch#2885
* Return HTTP 400 for innerkey/matrix stats aggs
* Add integration test for unimplemented matrix aggs
Original commit: elastic/x-pack-elasticsearch@34459c59aa
Fixes the error message that SQL produces when it sees unsupported
indexes. It was always returning all broken indexes as "unknown" even
though we have much better error messages. It was just throwing them
away.
I caught this originally when backporting to 6.x where we had a test
that we produced a useful error message when the user attempted to run
SQL on an index with more than one type. We couldn't run that in the
feature/sql branch because it is inside the full cluster restart tests
and the "old" version of Elasticsearch used in those tests in
feature/sql is 6.x which doesn't allow indexes with multiple types. When
I backported to 6.x the test failed because it hadn't been run before.
In addition to fixing that test and the problem, this adds another test
that will reveal the problem when run in the feature/sql and master
branch.
Original commit: elastic/x-pack-elasticsearch@c7b787baee
Previously I'd added tests for JDBC and CLI that I *thought* used SSL but they didn't! I wasn't careful...
Testing changes:
* Actually enable SSL/HTTPS in the `qa:sql:security:ssl` subproject.
* Rework how `RemoteCli` handles security. This allows us to configure SSL, the keystore, and the username and password in a much less error prone way.
* Fix up JDBC tests to properly use SSL.
* Allow the `CliFixture` to specify the keystore location.
* Switch `CliFixture` and `RemoteCli` from sending the password in the connection string to filling out the prompt for it.
* Have `CliFixture` also send the keystore password when a keystore is configured.
This makes the following production code changes:
* Allow the CLI to configure the keystore location with the `-k`/`-keystore_location` parameters.
* If the keystore location is configured then the CLI will prompt for the password.
* Allow the configuration of urls starting with `https`.
* Improve the exception thrown when the URL doesn't parse by adding a suppressed exception with the original parse error, before we tried to add `http://` to the front of it.
Original commit: elastic/x-pack-elasticsearch@97fac4a3b4
This commits adds a new end point for closing in-flight cursors, it also ensures that all cursors are properly closed by adding after test checks that ensures that we don't leave any search context open.
relates elastic/x-pack-elasticsearch#2878
Original commit: elastic/x-pack-elasticsearch@1052ea28dc
This is mainly a promotion of Literal to Attribute to better handle folding expressions from extracted queries
Original commit: elastic/x-pack-elasticsearch@c3bb48bb61
Igor Motov
Costin Leau
Nik Everett
Lee Hinman
Costin Leau