Commit Graph

144 Commits

Author SHA1 Message Date
Nik Everett f92684c436 Fix SQL test
Original commit: elastic/x-pack-elasticsearch@14d07c2607
2017-10-27 17:38:34 -04:00
Costin Leau 660cb3b814 Upgrade ANTLR to 4.5.3
Relates to elastic/x-pack-elasticsearch#26834

Original commit: elastic/x-pack-elasticsearch@57ba8f1d5f
2017-10-27 23:29:00 +03:00
Costin Leau 8d5572a77b SQL: improve validation of properties (both names and values) (elastic/x-pack-elasticsearch#2759)
* refactor Configuration class

move away from Properties
perform validation of settings names and values at startup
better exception handling

Original commit: elastic/x-pack-elasticsearch@d8a9edeccf
2017-10-27 22:54:03 +03:00
Costin Leau af591b9edd SQL: Remove usage of Settings inside SqlSettings (elastic/x-pack-elasticsearch#2757)
* Remove usage of Settings inside SqlSettings

Also hook client timeouts to the backend
Set UTC as default timezone when using CSV
As the JVM timezone changes, make sure to pin it to UTC since this is what the results are computed against

Original commit: elastic/x-pack-elasticsearch@3e7aad8c1f
2017-10-27 18:55:59 +03:00
Costin Leau 6b1d0d1c8e SQL: Remove time threshold for rule executor (elastic/x-pack-elasticsearch#2750)
Refactor RuleExecutor to eliminate time-based thresholds

Original commit: elastic/x-pack-elasticsearch@ba131f8058
2017-10-27 18:43:13 +03:00
Igor Motov 387944d176 SQL: Move special joda time handling into DocValueExtractor (elastic/x-pack-elasticsearch#2758)
Moves joda time handling into DocValueExtractor, that's the only place where it occurs at the moment.

Original commit: elastic/x-pack-elasticsearch@205e82990a
2017-10-22 15:54:00 +02:00
Costin Leau 353c0c500b Fix and enable datetime tests (elastic/x-pack-elasticsearch#2680)
Firstly, data in H2 is now stored in TIMESTAMP WITH TIME ZONE since H2
does not allow a global TZ to be set and picks the JVM TZ when a record
is read.
JdbcAssert is now aware of this allows TIMESTAMP with TZ == TIMESTAMP

Discovered a serious bug in DateTimeFunction - unfortunately date
histogram is not useful except for year since most extract functions
avoid ordering which a histogram preserves.
Thus most DTF are now terms aggs with scripting.

Improved a bug that caused duplicate functions to not be detected because
of aliasing.

Moved some datetime tests to CSV but the aggs tests now are in sync with
H2

Fixed bug that caused arithmetic on aggs to not be properly resolved by
splitting the processor definition tree to aggName (unresolved) and
aggPath (resolved)

Original commit: elastic/x-pack-elasticsearch@e75ada68f1
2017-10-13 13:52:48 +03:00
Costin Leau 2bbb86eff7 Eliminate NOCOMMITS per discussion
Original commit: elastic/x-pack-elasticsearch@bd091a6608
2017-10-13 00:33:16 +03:00
Nik Everett 852af7de57 Fix error handling in SQL's CLI (elastic/x-pack-elasticsearch#2730)
We weren't returning errors correctly from the server
or catching them correctly in the CLI. This fixes that
and adds simple integration tests.

Original commit: elastic/x-pack-elasticsearch@259da0da6f
2017-10-12 16:32:15 +00:00
Nik Everett 6478713304 Add support for username and password in SQL CLI (elastic/x-pack-elasticsearch#2718)
Add support for username and password in SQL CLI and adds tests that CLI works with security.

Original commit: elastic/x-pack-elasticsearch@39c8dbfc97
2017-10-12 15:55:29 +00:00
Costin Leau fa4504ed28 Fix some NOCOMMITs
Original commit: elastic/x-pack-elasticsearch@1a6ac1e6c6
2017-10-12 14:24:56 +03:00
Costin Leau e3d072aeea Make JDBC driver throw only SQLException (elastic/x-pack-elasticsearch#2708)
A JDBC driver should throw only checked SQLExceptions.
Introduce JdbcSQLException and fix some no-commits along the way.

Original commit: elastic/x-pack-elasticsearch@299fcf9ace
2017-10-12 12:27:19 +03:00
Igor Motov 7f0c44f138 SQL: Fix string to boolean conversion (elastic/x-pack-elasticsearch#2728)
This commit also adds tests for conversion to boolean

Original commit: elastic/x-pack-elasticsearch@d7740929a2
2017-10-11 19:56:36 -04:00
Igor Motov 06b4c043b9 SQL: Add toXcontent test for SqlResponse (elastic/x-pack-elasticsearch#2726)
Original commit: elastic/x-pack-elasticsearch@aa7b8f1a91
2017-10-11 19:55:59 -04:00
Costin Leau 2026198dd4 Add size for column tests (elastic/x-pack-elasticsearch#2685)
Add displaySize to columnInfo

Original commit: elastic/x-pack-elasticsearch@ed1d265e98
2017-10-12 00:03:41 +03:00
Costin Leau c9a41e111e Improve Sql exception reporting (elastic/x-pack-elasticsearch#2679)
Better handling of SQL exceptions (result of incorrect queries) vs
unexpected ones (engine failure, ES...)

Original commit: elastic/x-pack-elasticsearch@2698402cdb
2017-10-11 23:23:52 +03:00
Costin Leau 57fcbb81cb SQL: Improve JDBC communication (elastic/x-pack-elasticsearch#2660)
* Improve JDBC communication
Jdbc HTTP client uses only one url for messages and relies on / for ping
Fixed ES prefix being discarded (missing /)
Add HEAD handler for JDBC endpoint

Original commit: elastic/x-pack-elasticsearch@389f82262e
2017-10-11 23:03:03 +03:00
Costin Leau 1cd6fb23ec Make SearchCursor limit aware
Original commit: elastic/x-pack-elasticsearch@c4839bc293
2017-10-10 19:22:42 +03:00
Igor Motov ed712d0e3f SQL: clean unused Exceptions and replace Util functions with core versions (elastic/x-pack-elasticsearch#2654)
This commit removes ThrowableConsumer, WrappingException, ActionUtils and ObjectUtils by replacing them with core equivalents when needed.

Original commit: elastic/x-pack-elasticsearch@5a68418a3d
2017-10-09 18:00:29 -04:00
Igor Motov fa410095ce SQL: remove synchronous execute method from Command (elastic/x-pack-elasticsearch#2653)
We switched most of the commands to asynchronous mode, this commit removes the synchronous version of exec to prevent accidental use of it in the future.

Original commit: elastic/x-pack-elasticsearch@03d0a6350d
2017-10-09 16:01:56 -04:00
Nik Everett 8675f6998b Use LF line endings in SQL generated files (elastic/x-pack-elasticsearch#26822) (elastic/x-pack-elasticsearch#2706)
Right now if you run `gradle regen` on Windows you'll get `CRLF` line
endings on all the ANTLR generated files because we run
```
ant.fixcrlf(srcdir: outputPath) {
    patternset(includes: 'SqlBase*.java')
}
```

The docs for fixcrlf say that the default line endings that it
corrects to is based on the OS:
https://ant.apache.org/manual/Tasks/fixcrlf.html

This change locks it to `LF`.

Original commit: elastic/x-pack-elasticsearch@4396729e04
2017-10-07 07:54:21 -04:00
Costin Leau 00405ddace Fix incorrect getColumnClassName
Original commit: elastic/x-pack-elasticsearch@a11bfd477f
2017-10-05 14:12:10 +03:00
Costin Leau e1a7c59d15 Rework StringUtilsTest from plain JUnit to ESTestCase
Original commit: elastic/x-pack-elasticsearch@cd16043f23
2017-09-30 01:17:33 +03:00
Costin Leau e0d02033de More JDBC improvements
properly return the precision for VARCHAR
ignore type when specified in index pattern

Original commit: elastic/x-pack-elasticsearch@71a5ac1812
2017-09-30 00:40:57 +03:00
Costin Leau cc66bbaa00 All jdbc client escaping is done on the server
Original commit: elastic/x-pack-elasticsearch@2b8b7c8c2e
2017-09-29 20:09:58 +03:00
Costin Leau 748f9f3cd0 Fixes two NOCOMMITs
The DriverManager registration is now public so the user can control it
(static block might not be enough). The checked exception is also
logged and the rest rethrown.

Fixed param type name to return the correct value

Original commit: elastic/x-pack-elasticsearch@026476a6e4
2017-09-29 16:19:48 +03:00
Costin Leau f8776b8d43 Add more comments
Original commit: elastic/x-pack-elasticsearch@3be202947b
2017-09-29 15:21:48 +03:00
Costin Leau 1824d4771c fix incorrect column info returned by metadata
Original commit: elastic/x-pack-elasticsearch@dfda0e498c
2017-09-29 15:21:39 +03:00
Nik Everett c8e69b160e SQL: Fix build
Fix a few forgetten things from the validation change.

Original commit: elastic/x-pack-elasticsearch@807098dc6a
2017-09-28 16:57:58 -04:00
Costin Leau d634314dd1 Add comments to SqlSession
Original commit: elastic/x-pack-elasticsearch@82291d41c8
2017-09-28 22:29:23 +03:00
Costin Leau aca8a5b6c0 Analysis validation (elastic/x-pack-elasticsearch#2651)
Rework unresolved items messages
Update URLs in embedded HttpServers
Add antlr-runtime for embedded classpath

Original commit: elastic/x-pack-elasticsearch@36f0331d90
2017-09-28 22:27:20 +03:00
Igor Motov 33717f477f SQL: Make ShowColumns async (elastic/x-pack-elasticsearch#2638)
Original commit: elastic/x-pack-elasticsearch@54214b1109
2017-09-28 14:22:22 -04:00
Nik Everett 6be886edd8 Drop SQL's Session features (elastic/x-pack-elasticsearch#2639)
They don't work now and don't seem important enough to fix for
the first release. Maybe not ever.

Original commit: elastic/x-pack-elasticsearch@61e2b77e08
2017-09-28 13:41:58 -04:00
Igor Motov 6bae50a228 SQL: JdbcException clean up (elastic/x-pack-elasticsearch#2637)
Original commit: elastic/x-pack-elasticsearch@bcd0c59b37
2017-09-27 15:22:05 -04:00
Costin Leau c33dfe7dbe Hook _translate rest endpoint
Add rest testcase plus fix some NOCOMMITs

Original commit: elastic/x-pack-elasticsearch@150576869c
2017-09-27 18:50:35 +03:00
Nik Everett 4da12381bf Remove RowSetCursor (elastic/x-pack-elasticsearch#2624)
`RowSetCursor` was just like `RowSet` only it had methods that allowed
you to scroll to the next page. We now use `RowSet#nextPageCursor` to
get the next page in a way that doesn't require us to store state on
the server. So we can remove `RowSetCursor` entirely now.

Original commit: elastic/x-pack-elasticsearch@6a4a1efb20
2017-09-25 17:35:03 -04:00
Nik Everett c7c79bc1c0 Add scrolling support to jdbc (elastic/x-pack-elasticsearch#2524)
* Switch `ResultSet#getFetchSize` from returning the *requested*
fetch size to returning the size of the current page of results.
For scrolling searches without parent/child this'll match the
requested fetch size but for other things it won't. The nice thing
about this is that it lets us tell users a thing to look at if
they are wondering why they are using a bunch of memory.
* Remove all the entire JDBC action and implement it on the REST
layer instead.
* Create some code that can be shared between the cli and jdbc
actions to properly handle errors and request deserialization so
there isn't as much copy and paste between the two. This helps
because it calls out explicitly the places where they are different.
  * I have not moved the CLI REST handle to shared code because
I think it'd be clearer to make that change in a followup PR.
* There is now no more need for constructs that allow iterating
all of the results in the same request. I have not removed these
because I feel that that cleanup is big enough to deserve its own
PR.

Original commit: elastic/x-pack-elasticsearch@3b12afd11c
2017-09-25 14:41:46 -04:00
Nik Everett 63a5ad0b1d Add versioned DataInput and DataOutput to SQL (elastic/x-pack-elasticsearch#2600)
These wrap `DataInput` and `DataOutput` to add the protocol
version being serialized. This is similar to the mechanism
used by core and it has made adding and removing fields from
the serialization protocol fairly simple.

Original commit: elastic/x-pack-elasticsearch@90b3f1199a
2017-09-25 11:01:57 -04:00
Igor Motov 3ca00353c8 SQL: Clean up unused calls in CLI (elastic/x-pack-elasticsearch#2593)
Original commit: elastic/x-pack-elasticsearch@ba3c64b09c
2017-09-22 10:59:55 -04:00
Nik Everett 3b8d8487f2 Sql: Consistent terminal setup for CLI
The CI tests are failing when everything works locally. It *looks*
like we are running the CLI in using autodetect mode and I expect
that Jenkins doens't *have* a terminal so It'll autodetect to
`dumb` which doesn't output encoding.

Original commit: elastic/x-pack-elasticsearch@a9075648a2
2017-09-21 17:44:56 -04:00
Nik Everett 71a33323ff SQL: Soften some more NOCOMMITs
Original commit: elastic/x-pack-elasticsearch@910e2485df
2017-09-21 11:33:30 -04:00
Nik Everett 8a05c1b81f Move all sql integration tests into qa (elastic/x-pack-elasticsearch#2432)
Builds on elastic/x-pack-elasticsearch#2403 to move all of sql's integration testing into
qa modules with different running server configurations. The
big advantage of this is that it allows us to test the cli and
jdbc with security present.

Creating a project that depends on both cli and jdbc and the
server has some prickly jar hell issues because cli and jdbc
package their dependencies in the jar. This works around it
in a few days:
1. Include only a single copy of the JDBC dependencies with
careful gradle work.
2. Do not include the CLI on the classpath at all and instead
run it externally.

I say "run it externally" rather than "fork it" because Elasticsearch
tests aren't allowed to fork other processes. This is forbidden
by seccomp on linux and seatbelt on osx and cannot be explicitly
requested like additional security manager settings. So instead
of forking the CLI process directly the tests interact with a test
fixture that isn't bound by Elasticsearch's rules and *can* fork
it.

This forking of the CLI has a nice side effect: it forces us to
make sure that things like security and connection strings other
than `localhost:9200` work. The old test could and did work around
missing features like that. The new tests cannot so I added the
ability to set the connection string. Configuring usernames and
passwords was also not supported but I did not add support for
that, only created the failing test and marked it as `@AwaitsFix`.

Original commit: elastic/x-pack-elasticsearch@560c6815e3
2017-09-21 09:58:52 -04:00
Costin Leau 6cc3c067b7 Introduce _sql/translate endpoint
Builds on RestSqlAction and in fact, extends SqlAction to keep up
with future request settings.

Original commit: elastic/x-pack-elasticsearch@7bbef4bdff
2017-09-20 21:48:53 +03:00
Nik Everett 1405773acb Fix serialization for HitExtractorProcessor
Also get more information when SQL fails in IT.

Original commit: elastic/x-pack-elasticsearch@09f6625274
2017-09-20 14:15:35 -04:00
Costin Leau 87293272d8 Arithmetic
* big refactor of Processor by introducing ProcessorDefinition an
immutable tree structure used for resolving multiple inputs across
folding (in particular for aggregations) which at runtime gets
translated into 'compiled' or small Processors

Add expression arithmetic, expression folding and type coercion

Folding
* for literals, scalars and inside the optimizer

Type validation happens per type hierarchy (numeric vs decimal) not type
Ceil/Floor/Round functions return long/int instead of double
ScalarFunction preserves ProcessorDefinition instead of functionId

Original commit: elastic/x-pack-elasticsearch@a703f8b455
2017-09-20 19:32:37 +03:00
Nik Everett b8f5720283 Switch SqlGetIndicesAction after merge
Merging picked up a change that starts to migrate away from Streamable
and we have to handle it.

Original commit: elastic/x-pack-elasticsearch@fa3489e128
2017-09-20 12:29:42 -04:00
Nik Everett 98029ecaf8 Re-add member mistakenly removed
Original commit: elastic/x-pack-elasticsearch@487369b74c
2017-09-20 12:00:34 -04:00
Nik Everett 6ea902f913 SQL: NOCOMMIT cleanup
Removes a few NOCOMMITs that are tracked other places and updates
a few with plans on how to work on them.

Original commit: elastic/x-pack-elasticsearch@8d1cfdf4ee
2017-09-20 11:19:05 -04:00
Nik Everett 1ede8cadf7 Ignore tabs in sql test keystore files (elastic/x-pack-elasticsearch#2544)
These are just tabs in binary data....

Original commit: elastic/x-pack-elasticsearch@bcfa3feec0
2017-09-19 09:32:42 -04:00
Nik Everett 76543491bc SQL: Soften NOCOMMIT to TODO
I'm adding this to the SQL tracker issue. It is more low priority
but we should think about doing it before merging because it is
fairly easy:

Exceptions have many constructors that take a string pattern and
arguments. These kinds of things have mostly been removed in core
because we prefer explicitly building the message at the call site
for clarity. We should probably remove these constructors because
it is easy and another way that we can be more like core quickly.

Original commit: elastic/x-pack-elasticsearch@fa4326c4d1
2017-09-18 18:07:59 -04:00