We weren't returning errors correctly from the server
or catching them correctly in the CLI. This fixes that
and adds simple integration tests.
Original commit: elastic/x-pack-elasticsearch@259da0da6f
The CI tests are failing when everything works locally. It *looks*
like we are running the CLI in using autodetect mode and I expect
that Jenkins doens't *have* a terminal so It'll autodetect to
`dumb` which doesn't output encoding.
Original commit: elastic/x-pack-elasticsearch@a9075648a2
Builds on elastic/x-pack-elasticsearch#2403 to move all of sql's integration testing into
qa modules with different running server configurations. The
big advantage of this is that it allows us to test the cli and
jdbc with security present.
Creating a project that depends on both cli and jdbc and the
server has some prickly jar hell issues because cli and jdbc
package their dependencies in the jar. This works around it
in a few days:
1. Include only a single copy of the JDBC dependencies with
careful gradle work.
2. Do not include the CLI on the classpath at all and instead
run it externally.
I say "run it externally" rather than "fork it" because Elasticsearch
tests aren't allowed to fork other processes. This is forbidden
by seccomp on linux and seatbelt on osx and cannot be explicitly
requested like additional security manager settings. So instead
of forking the CLI process directly the tests interact with a test
fixture that isn't bound by Elasticsearch's rules and *can* fork
it.
This forking of the CLI has a nice side effect: it forces us to
make sure that things like security and connection strings other
than `localhost:9200` work. The old test could and did work around
missing features like that. The new tests cannot so I added the
ability to set the connection string. Configuring usernames and
passwords was also not supported but I did not add support for
that, only created the failing test and marked it as `@AwaitsFix`.
Original commit: elastic/x-pack-elasticsearch@560c6815e3
* big refactor of Processor by introducing ProcessorDefinition an
immutable tree structure used for resolving multiple inputs across
folding (in particular for aggregations) which at runtime gets
translated into 'compiled' or small Processors
Add expression arithmetic, expression folding and type coercion
Folding
* for literals, scalars and inside the optimizer
Type validation happens per type hierarchy (numeric vs decimal) not type
Ceil/Floor/Round functions return long/int instead of double
ScalarFunction preserves ProcessorDefinition instead of functionId
Original commit: elastic/x-pack-elasticsearch@a703f8b455
Removes a few NOCOMMITs that are tracked other places and updates
a few with plans on how to work on them.
Original commit: elastic/x-pack-elasticsearch@8d1cfdf4ee
* Move CLI to TransportSqlAction
* Moves REST endpoint from `/_cli` to `/_sql/cli`
* Removes the special purpose CLI transport action instead
implements the CLI entirely on the REST layer, delegating
all SQL stuff to the same action that backs the `/_sql` REST
API.
* Reworks "embedded testing mode" to use a `FilterClient` to
bounce capture the sql transport action and execute in embedded.
* Switches CLI formatting from consuming the entire response
to consuming just the first page of the response and returning
a `cursor` that can be used to read the next page. That read is
not yet implemented.
* Switch CLI formatting from the consuming the `RowSetCursor` to
consuming the `SqlResponse` object.
* Adds tests for CLI formatting.
* Support next page in the cli
* Rename cli's CommandRequest/CommandResponse to
QueryInitRequest/QueryInitResponse to line up with jdbc
* Implement QueryPageRequest/QueryPageResponse in cli
* Use `byte[]` to represent the cursor in the cli. Those bytes
mean something, but only to the server. The only reasonint that
the client does about them is "if length == 0 then there isn't a
next page."
* Pull common code from jdbc's QueryInitRequest, QueryPageRequest,
QueryInitResponse, and QueryPageResponse into the shared-proto
project
* By implication this switches jdbc's QueryPageRequest to using
the same cursor implementation as the cli
Original commit: elastic/x-pack-elasticsearch@193586f1ee
This integrates SQL's metadata calls with security by creating
`SqlIndicesAction` and routing all of SQL's metadata calls through
it. Since it *does* know up from which indices it is working against
it can be an `IndicesRequest.Replaceable` and integrate with the
existing security infrastructure for filtering indices.
This request is implemented fairly similarly to the `GetIndexAction`
with the option to read from the master or from a local copy of
cluster state. Currently SQL forces it to run on the local copy
because the request doesn't properly support serialization. I'd
like to implement that in a followup.
Original commit: elastic/x-pack-elasticsearch@15f9512820
Indices discovery actively ignores indices with more than one type.
However queries made such indices throw an exception (assuming the user
by mistake or not, selects such an index).
Original commit: elastic/x-pack-elasticsearch@16855c7b8f
Too big. Sorry. Some good things though:
1. Share some code between CLI and JDBC. Probably a good thing
at this point, better as we go on, I think.
2. Add round trip tests for all of proto.
3. Remove the `data` member from `QueryInitResponse` and
`QueryPageResponse` so we response serialization is consistent with
everything else.
Original commit: elastic/x-pack-elasticsearch@c6940a32ed
* Move read from a static method to a ctor to mirror core.
* Make read and writes read and write the same data.
* Instead of the "header" integer use a byte for the response
type.
* For responses that do not make their request type obvious
then serialize the request type.
* Remove the request type member from requests and responses and
replace with an abstract method. These type members have caused
us trouble in core in the past.
* Remove the Message superclass as it didn't have anything in it.
* Pass client version to the request reader and response writer.
* Add round trip tests for the protocol.
* Force Requests and Responses to provide good `toString`, `equals`,
and `hashCode`.
Original commit: elastic/x-pack-elasticsearch@653ed8c27f
* Switch `data` member from Object to `String`
* Compress packages on server so easier to build `data` as `String`
* Move write of `data` member into `encode` method
* Move read of `data` member into ctor
Original commit: elastic/x-pack-elasticsearch@e3a52e7493
* A unit test for cli
* Licenses for cli
* Remove licenses for protos (no more deps)
* `SHOW TABLES` returns results in order (makes testing easier)
* Clean up embedded jdbc server
* Wire up embedded cli server
Original commit: elastic/x-pack-elasticsearch@b98aaf446b
`gradle check -xforbiddenPatterns` now passes in jdbc.
This makes running the embedded HTTP server slightly more difficult,
you now have to add the following to your jvm arguments.
```
-ea -Dtests.rest.cluster=localhost:9200 -Dtests.embed.sql=true -Dtests.security.manager=false
```
Depending on your environment the embedded jdbc connection may give
spurious failures that look like:
```
org.elasticsearch.xpack.sql.jdbc.jdbc.JdbcException: RemoteTransportException[[node-0][127.0.0.1:9300][indices:data/read/search]]; nested: SearchPhaseExecutionException[]; nested: GeneralScriptException[Failed to compile inline script [( params.a0 > params.v0 ) && ( params.a1 > params.v1 )] using lang [painless]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting];
...
Caused by: Failed to execute phase [fetch],
..
Caused by: GeneralScriptException[Failed to compile inline script [( params.a0 > params.v0 ) && ( params.a1 > params.v1 )] using lang [painless]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting];
...
Caused by: CircuitBreakingException[[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting]
```
`gradle check` works around this by setting `script.max_compilations_per_minute`
to `1000`.
Another change is that we no longer support loading the test data by
uncommenting some code. Instead we load the test data into Elaticsearch
before the first test and we deleted it after the last test. This is
so that tests that required different test data can interoperate with
eachother. The spec tests all use the same test data but the metadata
tests do not.
Original commit: elastic/x-pack-elasticsearch@8b8f684ac1
Costin Leau
Nik Everett
Igor Motov