Commit Graph

278 Commits

Author SHA1 Message Date
Igor Motov d5525f38f3 SQL: Return proper NUM_PREC_RADIX for non-numeric types (elastic/x-pack-elasticsearch#3086)
NUM_PREC_RADIX should be 10 for integer types, 2 for rational types and null for everything else.

relates elastic/x-pack-elasticsearch#3085

Original commit: elastic/x-pack-elasticsearch@81d5ee04b3
2017-11-29 08:21:42 -05:00
Nik Everett 193f22b97f SQL: Support larger strings in binary protocol (elastic/x-pack-elasticsearch#3130)
While we're fairly sure we're going to remove the binary protocol in the
long run, we're also fairly sure we're going to release the first
version of SQL with the binary protocol. One big problem with it is that
it blows up when it attempts to serialize fairly long strings. These
long strings are actually quite common in the CLI. They are also
possible in JDBC. I say "fairly long strings" because exactly how long
the strings has to be is kind of funky. It is based on the number of
bytes that it takes to encode the string, and the strings are encoded in
a utf-8-like encoding of utf-16 encoded string documented here:
https://docs.oracle.com/javase/7/docs/api/java/io/DataOutput.html#writeUTF(java.lang.String)

Anyway, this fixes the protocol for these "fairly long strings" by
chunking the strings and adding an extra 4 byte integer before each
string to count the number of chunks. After that 4 byte integer the
strings are serialized using the "normal" DataInput/DataOutput encoding,
the funny utf-8-like encoding of the utf-16 encoded string.

relates elastic/x-pack-elasticsearch#3018

Original commit: elastic/x-pack-elasticsearch@11f0d59f20
2017-11-27 18:59:31 -05:00
Nik Everett df802b40c8 SQL: Rework SSL testing (elastic/x-pack-elasticsearch#3126)
Drop the ssl tests against the java builtin https server. They were
failing and the failures were undebuggable. I still don't know what was
happening because you can't get any logging out of the server.

Add SSL tests against Elasticsearch because that is what actually needs
to work.

relates elastic/x-pack-elasticsearch#2870

Original commit: elastic/x-pack-elasticsearch@284cf7fb58
2017-11-27 18:52:16 -05:00
Igor Motov 5c88fa0b3b SQL: Add support for plain text output to /_sql endpoint (elastic/x-pack-elasticsearch#3124)
The /_sql endpoint now returns the results in the text format by default. Structured formats are also supported using the format parameter or accept header similar to _cat endpoints.

Original commit: elastic/x-pack-elasticsearch@4353793b83
2017-11-27 18:10:13 -05:00
Igor Motov 0228020c5c SQL: Switch to URI parsing (elastic/x-pack-elasticsearch#3108)
Replaces custom URL parsing with URI parsing and moves baseURI from each client call into connection configuration.

relates elastic/x-pack-elasticsearch#2882

Original commit: elastic/x-pack-elasticsearch@c51059f56f
2017-11-27 18:09:05 -05:00
Nik Everett f97f56ba54 SQL: Throw exceptions on errors (elastic/x-pack-elasticsearch#3066)
Instead of returning "error response" objects and then translating them
into SQL exceptions this just throws the SQL exceptions directly. This
means the CLI catches exceptions and prints out the messages which isn't
ideal if this were hot code but it isn't and this is a much simpler way
of doing things.

Original commit: elastic/x-pack-elasticsearch@08431d3941
2017-11-22 11:22:31 -05:00
Igor Motov a4915a5714 SQL: remove all remaining NOCOMMITs
relates elastic/x-pack-elasticsearch#2873

Original commit: elastic/x-pack-elasticsearch@68b206efd2
2017-11-21 14:37:59 -05:00
Igor Motov 2fe4da80ad SQL: add filter support in REST action (elastic/x-pack-elasticsearch#3045)
Adds the option to specify an elasticsearch filter in addition to the SQL query by introducing a filter parameter in the REST query which would create a boolean filter if the SQL query generates an elasticsearch query or a constant score query if SQL if the SQL query doesn't generates an elasticsearch query. Usage:

{
   "query": "SELECT * FROM index",
   "filter" : {  "term" : { "tag" : "tech" } }
}

relates elastic/x-pack-elasticsearch#2895

Original commit: elastic/x-pack-elasticsearch@9a73813c7f
2017-11-21 11:40:38 -05:00
Igor Motov 3e14c30aa1 SQL: Add version to cursor serialization (elastic/x-pack-elasticsearch#3064)
This commits also simplifies the serialization mechanism by remove 2 ways to serialize the cursor. Adding the version there was complicating things too much otherwise.

Original commit: elastic/x-pack-elasticsearch@4f2c541e0a
2017-11-21 11:24:27 -05:00
Igor Motov 318621cdc1 SQL: Move all SQL projects into org.elasticsearch.sql package (elastic/x-pack-elasticsearch#3069)
Elasticsearch plugins are traditionally using org.elasticsearch.plugin group, clients are using org.elasticsearch.client, so it makes sense to move all sql pieces and deliverables into org.elasticsearch.sql group. Otherwise, users might get really confused about an artifact named `server` in the group `org.elasticsearch`.

Original commit: elastic/x-pack-elasticsearch@c3a4042040
2017-11-21 10:53:40 +01:00
Nik Everett 87a3e93fae SQL: Improve error message for huge error responses (elastic/x-pack-elasticsearch#3049)
If the user has a funky proxy or something that throws back a massive
error message to the CLI or JDBC then return a sensible error message.

Original commit: elastic/x-pack-elasticsearch@94219969ea
2017-11-17 12:29:07 -05:00
Nik Everett fce5b494be SQL: Stop serializing errors for CLI and JDBC (elastic/x-pack-elasticsearch#3034)
Now that we can parse Elasticsearch's standard error messages in the CLI
and JDBC client we can just let those standard error messages bubble out
of Elasticsearch rather than catch and encode them.

In a followup we can remove the encoding entirely.

Original commit: elastic/x-pack-elasticsearch@bad043b6f7
2017-11-17 12:16:37 -05:00
Igor Motov de087bd859 SQL: Replace NOCOMMIT with TODO
Original commit: elastic/x-pack-elasticsearch@299f82db5c
2017-11-15 18:05:50 -05:00
Nik Everett adae4019dd SQL: improve exception when error can't be parsed (elastic/x-pack-elasticsearch#3024)
When an error response wasn't parseable we would through fairly obscure
error messages about what is wrong with it. Great if you are fixing a
bug in Elasticsearch, no great if a proxy eats the response. This adds
the response to the error messages so you can see what was returned.

Original commit: elastic/x-pack-elasticsearch@00e542afc8
2017-11-15 15:33:53 -05:00
Nik Everett 89e80e0cba Teach SQL to parse Elasticsearch's standard error responses (elastic/x-pack-elasticsearch#2764)
This teaches SQL to parse Elasticsearch's standard error responses
but doesn't change SQL to general Elasticsearch's standard error responses
in all cases. That can come in a followup. We do this parsing with
jackson-core, the same dependency Elasticsearch uses for parsing
json. We shade jackson-core in the JDBC driver so that users don't have to worry about
dependency clashes. We do not do so in the CLI because it is a standalone
application.

We get a few "bonus" changes along the way:
1. We save a copy operation. Before this change responses were spooled
into memory and then parsed. After this change they are parsed directly
from the response stream.
2. We had a few classes entirely to support the spooling operation that we
no longer need: `BytesArray`, `FastByteArrayInputStream`, and
`BasicByteArrayOutputStream`.
3. SQL's `Version` was incorrectly parsing the version from the jar manifest.
We didn't notice because the test was rigged to return `UNKNOWN` because
we *were* running the test from the compiled classes directory instead of the
jar. As part of shading jackson we moved running the tests to running against
the shaded jar. Now we can actually assert that we parse the version correctly.
It turns out we weren't. So I fixed it.

Original commit: elastic/x-pack-elasticsearch@2e8f397bf4
2017-11-14 21:31:35 -05:00
Costin Leau 94d0a2d1ee Polishing for handling subtleties in the JDBC behavior: (elastic/x-pack-elasticsearch#2967)
1. decouple JdbcDriver from other classes to not trigger static
initialization (this happens through JDBC service discovery)
2. reduce visibility of JdbcDriver#close so only on jar unloading it
gets triggered
3. mark 3 methods introduced in Jdbc 4.1 as unsupported (their semantics
are somewhat weird)
4. Move versioning info in one class
5. Hook Version class in both JDBC entry points to perform cp sanity
checks
6. Remove JdbcDataSource#close (DebugLog are closed when the Driver gets
unloaded by the DriverManager) as there can be multiple instances of DS
but only one for Driver known by the DriverManager

Replace Strings with constants
Properly set TZ in security tests as well
JdbcDataSource is more defensive with its internal properties
JdbcConfiguration password parameter is aligned with JDBC DriverManager
Remove usage of JdbcConnection API
Removed JdbcConnection#setTimeZone - this encourages folks to use our
private API which would tie us down.
It is somewhat limiting for folks but it has less downsides overall and
does not trip debugging (which adds a proxy unaware of this method).
Update docs
Add JdbcDataSource into the Jdbc suite

Original commit: elastic/x-pack-elasticsearch@c713665d53
2017-11-15 00:29:41 +02:00
Costin Leau c8eb883c47 Fix incorrect assertions and javadoc
Original commit: elastic/x-pack-elasticsearch@0e6ebc0a69
2017-11-14 09:46:26 +02:00
Costin Leau 9a0b43cd17 Fix several NOCOMMITS (elastic/x-pack-elasticsearch#2968)
* Fix several NOCOMMITS

- renamed Assert to Check to make the intent clear
- clarify esMajor/Minor inside connection (thse are actually our own
methods, not part of JDBC API)
- wire pageTimeout into Cursor#nextPage


Original commit: elastic/x-pack-elasticsearch@7626c0a44a
2017-11-14 01:08:10 +02:00
Igor Motov ea0e58f971 SQL: introduce setting to disable SQL (elastic/x-pack-elasticsearch#2966)
Adds xpack.sql.enabled setting to provide ability to disable SQL on elasticsearch nodes.

relates elastic/x-pack-elasticsearch#2872

Original commit: elastic/x-pack-elasticsearch@d13b72e9ea
2017-11-13 15:10:47 -05:00
Igor Motov 9b96c7eb33 SQL: clean unused classes and empty close() methods (elastic/x-pack-elasticsearch#2956)
relates elastic/x-pack-elasticsearch#2876

Original commit: elastic/x-pack-elasticsearch@c32c6a0058
2017-11-10 14:22:58 -05:00
Igor Motov 3c444c4719 SQL: Fix test when random JDK TZ doesn't exist in Joda (elastic/x-pack-elasticsearch#2903)
JodaTime timezone db can be out of date compared to that of the JDK which causes the JDBC Connection to fail when the randomized tests pick a timezone that's available in the JDK but not in Joda, like SystemV/PST8. This is happening because JdbcConnection configuration is using system default timezone and tries to pass it to Elasticsearch that is using joda. This commit, explicitly sets the time zone on JdbcConnection to a time zone randomly selected from a list of timezones that are known to both JDK and Joda.

relates elastic/x-pack-elasticsearch#2812

Original commit: elastic/x-pack-elasticsearch@b02e9794a8
2017-11-10 14:16:11 -05:00
Costin Leau 8f55a4192c Improve grammar so keywords have priority over identifiers
Original commit: elastic/x-pack-elasticsearch@8222e30060
2017-11-09 21:40:09 +02:00
Costin Leau c48d2b14dc Define structure for docs and improve grammar
The keywords inside SqlBase are now sorted alphabetically - much easier
to read and update the docs

Original commit: elastic/x-pack-elasticsearch@5aa89c5950
2017-11-09 17:49:10 +02:00
Luca Cavanna 753a8d6a6d Build: Run assemble automatically before running cli from gradle (elastic/x-pack-elasticsearch#2918)
Original commit: elastic/x-pack-elasticsearch@43f7396764
2017-11-07 20:51:07 +01:00
Nik Everett a211077554 Switch JDBC metadata to indicate all columns nullable (elastic/x-pack-elasticsearch#2835)
We were returning "nullability unknown" but in Elasticsearch all columns
are nullable.

Original commit: elastic/x-pack-elasticsearch@6ceae418ea
2017-11-04 23:28:49 +00:00
Nik Everett 6ce140cf0f SQL: Make schema optional in RowSet (elastic/x-pack-elasticsearch#2834)
This prevents us from having to hack a fake schema together on the
second (and third and fourth, etc) page of results.

Original commit: elastic/x-pack-elasticsearch@7ba3119daa
2017-11-04 22:52:13 +00:00
Nik Everett 44dc98385a SQL: Reuse a registry meant to be shared (elastic/x-pack-elasticsearch#2838)
The CLI and JDBC were meant to share the same named xcontent registry
but the cli action didn't reuse it by mistake.

Original commit: elastic/x-pack-elasticsearch@6c9af5a22a
2017-11-04 13:10:35 +00:00
Nik Everett 68779a81e4 SQL: Remove now unused class (elastic/x-pack-elasticsearch#2839)
The `AbstractSqlServer` class was used when SQL's CLI and JDBC were
transport actions. They are REST layer concepts now and it is unused so
it should go.

Original commit: elastic/x-pack-elasticsearch@25ec699564
2017-11-04 00:01:15 +00:00
Nik Everett 41284cae93 SQL: Fix the name of the indices lookup action (elastic/x-pack-elasticsearch#2840)
Fix the name of the action the SQL uses to lookup index information from
the cluster state. The old name was silly.

Original commit: elastic/x-pack-elasticsearch@805fb29662
2017-11-03 23:37:22 +00:00
Nik Everett 25a0ec42f6 SQL: Reorganize the translate action (elastic/x-pack-elasticsearch#2841)
Organizes the SQL translate action to match the way that x-pack has been
organizing new actions for a while. All of the pieces are put into the
same class file.

Original commit: elastic/x-pack-elasticsearch@def911c0ab
2017-11-03 23:25:52 +00:00
Nik Everett 43b3a7d7aa Fix import
I missed it in the last PR.

Original commit: elastic/x-pack-elasticsearch@58b2047165
2017-10-30 19:01:15 -04:00
Nik Everett c13f0a8272 Rename SQL's net-client project to shared-client (elastic/x-pack-elasticsearch#2756)
The `net-client` project contained more then just the `net-client`.
It contains stuff like `SuppressForbidden` and `Strings` and `IOUtil`
and other things shared between the CLI and JDBC. It also does contain
the http client. Anyway, it makes more sense to call it `shared-client`,
I think.

Alos updated the copywrite dates on the files that I touched because
they are all 2017 files.

Removed some uses of `String.EMPTY` because they don't buy us anything
and require an extra import. `""` is just one less step.

Original commit: elastic/x-pack-elasticsearch@465c6445c4
2017-10-30 22:59:01 +00:00
Nik Everett d933b1b48b Initial docs for SQL (elastic/x-pack-elasticsearch#2810)
Adds docs for the REST API, translate API, the CLI, and JDBC.

Next we need to add more example queries and documentation for our
extensions.

Original commit: elastic/x-pack-elasticsearch@ed6d1360d2
2017-10-30 17:23:27 +00:00
Costin Leau 66719f7a92 Improve DateTime script template
When dealing with TZ, the script is reused across extractors by
externalizing the tz & chronofield

Original commit: elastic/x-pack-elasticsearch@9e63ea8cd8
2017-10-28 13:35:16 +03:00
Nik Everett f92684c436 Fix SQL test
Original commit: elastic/x-pack-elasticsearch@14d07c2607
2017-10-27 17:38:34 -04:00
Costin Leau 660cb3b814 Upgrade ANTLR to 4.5.3
Relates to elastic/x-pack-elasticsearch#26834

Original commit: elastic/x-pack-elasticsearch@57ba8f1d5f
2017-10-27 23:29:00 +03:00
Costin Leau 8d5572a77b SQL: improve validation of properties (both names and values) (elastic/x-pack-elasticsearch#2759)
* refactor Configuration class

move away from Properties
perform validation of settings names and values at startup
better exception handling

Original commit: elastic/x-pack-elasticsearch@d8a9edeccf
2017-10-27 22:54:03 +03:00
Costin Leau af591b9edd SQL: Remove usage of Settings inside SqlSettings (elastic/x-pack-elasticsearch#2757)
* Remove usage of Settings inside SqlSettings

Also hook client timeouts to the backend
Set UTC as default timezone when using CSV
As the JVM timezone changes, make sure to pin it to UTC since this is what the results are computed against

Original commit: elastic/x-pack-elasticsearch@3e7aad8c1f
2017-10-27 18:55:59 +03:00
Costin Leau 6b1d0d1c8e SQL: Remove time threshold for rule executor (elastic/x-pack-elasticsearch#2750)
Refactor RuleExecutor to eliminate time-based thresholds

Original commit: elastic/x-pack-elasticsearch@ba131f8058
2017-10-27 18:43:13 +03:00
Igor Motov 387944d176 SQL: Move special joda time handling into DocValueExtractor (elastic/x-pack-elasticsearch#2758)
Moves joda time handling into DocValueExtractor, that's the only place where it occurs at the moment.

Original commit: elastic/x-pack-elasticsearch@205e82990a
2017-10-22 15:54:00 +02:00
Costin Leau 353c0c500b Fix and enable datetime tests (elastic/x-pack-elasticsearch#2680)
Firstly, data in H2 is now stored in TIMESTAMP WITH TIME ZONE since H2
does not allow a global TZ to be set and picks the JVM TZ when a record
is read.
JdbcAssert is now aware of this allows TIMESTAMP with TZ == TIMESTAMP

Discovered a serious bug in DateTimeFunction - unfortunately date
histogram is not useful except for year since most extract functions
avoid ordering which a histogram preserves.
Thus most DTF are now terms aggs with scripting.

Improved a bug that caused duplicate functions to not be detected because
of aliasing.

Moved some datetime tests to CSV but the aggs tests now are in sync with
H2

Fixed bug that caused arithmetic on aggs to not be properly resolved by
splitting the processor definition tree to aggName (unresolved) and
aggPath (resolved)

Original commit: elastic/x-pack-elasticsearch@e75ada68f1
2017-10-13 13:52:48 +03:00
Costin Leau 2bbb86eff7 Eliminate NOCOMMITS per discussion
Original commit: elastic/x-pack-elasticsearch@bd091a6608
2017-10-13 00:33:16 +03:00
Nik Everett 852af7de57 Fix error handling in SQL's CLI (elastic/x-pack-elasticsearch#2730)
We weren't returning errors correctly from the server
or catching them correctly in the CLI. This fixes that
and adds simple integration tests.

Original commit: elastic/x-pack-elasticsearch@259da0da6f
2017-10-12 16:32:15 +00:00
Nik Everett 6478713304 Add support for username and password in SQL CLI (elastic/x-pack-elasticsearch#2718)
Add support for username and password in SQL CLI and adds tests that CLI works with security.

Original commit: elastic/x-pack-elasticsearch@39c8dbfc97
2017-10-12 15:55:29 +00:00
Costin Leau fa4504ed28 Fix some NOCOMMITs
Original commit: elastic/x-pack-elasticsearch@1a6ac1e6c6
2017-10-12 14:24:56 +03:00
Costin Leau e3d072aeea Make JDBC driver throw only SQLException (elastic/x-pack-elasticsearch#2708)
A JDBC driver should throw only checked SQLExceptions.
Introduce JdbcSQLException and fix some no-commits along the way.

Original commit: elastic/x-pack-elasticsearch@299fcf9ace
2017-10-12 12:27:19 +03:00
Igor Motov 7f0c44f138 SQL: Fix string to boolean conversion (elastic/x-pack-elasticsearch#2728)
This commit also adds tests for conversion to boolean

Original commit: elastic/x-pack-elasticsearch@d7740929a2
2017-10-11 19:56:36 -04:00
Igor Motov 06b4c043b9 SQL: Add toXcontent test for SqlResponse (elastic/x-pack-elasticsearch#2726)
Original commit: elastic/x-pack-elasticsearch@aa7b8f1a91
2017-10-11 19:55:59 -04:00
Costin Leau 2026198dd4 Add size for column tests (elastic/x-pack-elasticsearch#2685)
Add displaySize to columnInfo

Original commit: elastic/x-pack-elasticsearch@ed1d265e98
2017-10-12 00:03:41 +03:00
Costin Leau c9a41e111e Improve Sql exception reporting (elastic/x-pack-elasticsearch#2679)
Better handling of SQL exceptions (result of incorrect queries) vs
unexpected ones (engine failure, ES...)

Original commit: elastic/x-pack-elasticsearch@2698402cdb
2017-10-11 23:23:52 +03:00
Costin Leau 57fcbb81cb SQL: Improve JDBC communication (elastic/x-pack-elasticsearch#2660)
* Improve JDBC communication
Jdbc HTTP client uses only one url for messages and relies on / for ping
Fixed ES prefix being discarded (missing /)
Add HEAD handler for JDBC endpoint

Original commit: elastic/x-pack-elasticsearch@389f82262e
2017-10-11 23:03:03 +03:00
Costin Leau 1cd6fb23ec Make SearchCursor limit aware
Original commit: elastic/x-pack-elasticsearch@c4839bc293
2017-10-10 19:22:42 +03:00
Igor Motov ed712d0e3f SQL: clean unused Exceptions and replace Util functions with core versions (elastic/x-pack-elasticsearch#2654)
This commit removes ThrowableConsumer, WrappingException, ActionUtils and ObjectUtils by replacing them with core equivalents when needed.

Original commit: elastic/x-pack-elasticsearch@5a68418a3d
2017-10-09 18:00:29 -04:00
Igor Motov fa410095ce SQL: remove synchronous execute method from Command (elastic/x-pack-elasticsearch#2653)
We switched most of the commands to asynchronous mode, this commit removes the synchronous version of exec to prevent accidental use of it in the future.

Original commit: elastic/x-pack-elasticsearch@03d0a6350d
2017-10-09 16:01:56 -04:00
Nik Everett 8675f6998b Use LF line endings in SQL generated files (elastic/x-pack-elasticsearch#26822) (elastic/x-pack-elasticsearch#2706)
Right now if you run `gradle regen` on Windows you'll get `CRLF` line
endings on all the ANTLR generated files because we run
```
ant.fixcrlf(srcdir: outputPath) {
    patternset(includes: 'SqlBase*.java')
}
```

The docs for fixcrlf say that the default line endings that it
corrects to is based on the OS:
https://ant.apache.org/manual/Tasks/fixcrlf.html

This change locks it to `LF`.

Original commit: elastic/x-pack-elasticsearch@4396729e04
2017-10-07 07:54:21 -04:00
Costin Leau 00405ddace Fix incorrect getColumnClassName
Original commit: elastic/x-pack-elasticsearch@a11bfd477f
2017-10-05 14:12:10 +03:00
Costin Leau e1a7c59d15 Rework StringUtilsTest from plain JUnit to ESTestCase
Original commit: elastic/x-pack-elasticsearch@cd16043f23
2017-09-30 01:17:33 +03:00
Costin Leau e0d02033de More JDBC improvements
properly return the precision for VARCHAR
ignore type when specified in index pattern

Original commit: elastic/x-pack-elasticsearch@71a5ac1812
2017-09-30 00:40:57 +03:00
Costin Leau cc66bbaa00 All jdbc client escaping is done on the server
Original commit: elastic/x-pack-elasticsearch@2b8b7c8c2e
2017-09-29 20:09:58 +03:00
Costin Leau 748f9f3cd0 Fixes two NOCOMMITs
The DriverManager registration is now public so the user can control it
(static block might not be enough). The checked exception is also
logged and the rest rethrown.

Fixed param type name to return the correct value

Original commit: elastic/x-pack-elasticsearch@026476a6e4
2017-09-29 16:19:48 +03:00
Costin Leau f8776b8d43 Add more comments
Original commit: elastic/x-pack-elasticsearch@3be202947b
2017-09-29 15:21:48 +03:00
Costin Leau 1824d4771c fix incorrect column info returned by metadata
Original commit: elastic/x-pack-elasticsearch@dfda0e498c
2017-09-29 15:21:39 +03:00
Nik Everett c8e69b160e SQL: Fix build
Fix a few forgetten things from the validation change.

Original commit: elastic/x-pack-elasticsearch@807098dc6a
2017-09-28 16:57:58 -04:00
Costin Leau d634314dd1 Add comments to SqlSession
Original commit: elastic/x-pack-elasticsearch@82291d41c8
2017-09-28 22:29:23 +03:00
Costin Leau aca8a5b6c0 Analysis validation (elastic/x-pack-elasticsearch#2651)
Rework unresolved items messages
Update URLs in embedded HttpServers
Add antlr-runtime for embedded classpath

Original commit: elastic/x-pack-elasticsearch@36f0331d90
2017-09-28 22:27:20 +03:00
Igor Motov 33717f477f SQL: Make ShowColumns async (elastic/x-pack-elasticsearch#2638)
Original commit: elastic/x-pack-elasticsearch@54214b1109
2017-09-28 14:22:22 -04:00
Nik Everett 6be886edd8 Drop SQL's Session features (elastic/x-pack-elasticsearch#2639)
They don't work now and don't seem important enough to fix for
the first release. Maybe not ever.

Original commit: elastic/x-pack-elasticsearch@61e2b77e08
2017-09-28 13:41:58 -04:00
Igor Motov 6bae50a228 SQL: JdbcException clean up (elastic/x-pack-elasticsearch#2637)
Original commit: elastic/x-pack-elasticsearch@bcd0c59b37
2017-09-27 15:22:05 -04:00
Costin Leau c33dfe7dbe Hook _translate rest endpoint
Add rest testcase plus fix some NOCOMMITs

Original commit: elastic/x-pack-elasticsearch@150576869c
2017-09-27 18:50:35 +03:00
Nik Everett 4da12381bf Remove RowSetCursor (elastic/x-pack-elasticsearch#2624)
`RowSetCursor` was just like `RowSet` only it had methods that allowed
you to scroll to the next page. We now use `RowSet#nextPageCursor` to
get the next page in a way that doesn't require us to store state on
the server. So we can remove `RowSetCursor` entirely now.

Original commit: elastic/x-pack-elasticsearch@6a4a1efb20
2017-09-25 17:35:03 -04:00
Nik Everett c7c79bc1c0 Add scrolling support to jdbc (elastic/x-pack-elasticsearch#2524)
* Switch `ResultSet#getFetchSize` from returning the *requested*
fetch size to returning the size of the current page of results.
For scrolling searches without parent/child this'll match the
requested fetch size but for other things it won't. The nice thing
about this is that it lets us tell users a thing to look at if
they are wondering why they are using a bunch of memory.
* Remove all the entire JDBC action and implement it on the REST
layer instead.
* Create some code that can be shared between the cli and jdbc
actions to properly handle errors and request deserialization so
there isn't as much copy and paste between the two. This helps
because it calls out explicitly the places where they are different.
  * I have not moved the CLI REST handle to shared code because
I think it'd be clearer to make that change in a followup PR.
* There is now no more need for constructs that allow iterating
all of the results in the same request. I have not removed these
because I feel that that cleanup is big enough to deserve its own
PR.

Original commit: elastic/x-pack-elasticsearch@3b12afd11c
2017-09-25 14:41:46 -04:00
Nik Everett 63a5ad0b1d Add versioned DataInput and DataOutput to SQL (elastic/x-pack-elasticsearch#2600)
These wrap `DataInput` and `DataOutput` to add the protocol
version being serialized. This is similar to the mechanism
used by core and it has made adding and removing fields from
the serialization protocol fairly simple.

Original commit: elastic/x-pack-elasticsearch@90b3f1199a
2017-09-25 11:01:57 -04:00
Igor Motov 3ca00353c8 SQL: Clean up unused calls in CLI (elastic/x-pack-elasticsearch#2593)
Original commit: elastic/x-pack-elasticsearch@ba3c64b09c
2017-09-22 10:59:55 -04:00
Nik Everett 3b8d8487f2 Sql: Consistent terminal setup for CLI
The CI tests are failing when everything works locally. It *looks*
like we are running the CLI in using autodetect mode and I expect
that Jenkins doens't *have* a terminal so It'll autodetect to
`dumb` which doesn't output encoding.

Original commit: elastic/x-pack-elasticsearch@a9075648a2
2017-09-21 17:44:56 -04:00
Nik Everett 71a33323ff SQL: Soften some more NOCOMMITs
Original commit: elastic/x-pack-elasticsearch@910e2485df
2017-09-21 11:33:30 -04:00
Nik Everett 8a05c1b81f Move all sql integration tests into qa (elastic/x-pack-elasticsearch#2432)
Builds on elastic/x-pack-elasticsearch#2403 to move all of sql's integration testing into
qa modules with different running server configurations. The
big advantage of this is that it allows us to test the cli and
jdbc with security present.

Creating a project that depends on both cli and jdbc and the
server has some prickly jar hell issues because cli and jdbc
package their dependencies in the jar. This works around it
in a few days:
1. Include only a single copy of the JDBC dependencies with
careful gradle work.
2. Do not include the CLI on the classpath at all and instead
run it externally.

I say "run it externally" rather than "fork it" because Elasticsearch
tests aren't allowed to fork other processes. This is forbidden
by seccomp on linux and seatbelt on osx and cannot be explicitly
requested like additional security manager settings. So instead
of forking the CLI process directly the tests interact with a test
fixture that isn't bound by Elasticsearch's rules and *can* fork
it.

This forking of the CLI has a nice side effect: it forces us to
make sure that things like security and connection strings other
than `localhost:9200` work. The old test could and did work around
missing features like that. The new tests cannot so I added the
ability to set the connection string. Configuring usernames and
passwords was also not supported but I did not add support for
that, only created the failing test and marked it as `@AwaitsFix`.

Original commit: elastic/x-pack-elasticsearch@560c6815e3
2017-09-21 09:58:52 -04:00
Costin Leau 6cc3c067b7 Introduce _sql/translate endpoint
Builds on RestSqlAction and in fact, extends SqlAction to keep up
with future request settings.

Original commit: elastic/x-pack-elasticsearch@7bbef4bdff
2017-09-20 21:48:53 +03:00
Nik Everett 1405773acb Fix serialization for HitExtractorProcessor
Also get more information when SQL fails in IT.

Original commit: elastic/x-pack-elasticsearch@09f6625274
2017-09-20 14:15:35 -04:00
Costin Leau 87293272d8 Arithmetic
* big refactor of Processor by introducing ProcessorDefinition an
immutable tree structure used for resolving multiple inputs across
folding (in particular for aggregations) which at runtime gets
translated into 'compiled' or small Processors

Add expression arithmetic, expression folding and type coercion

Folding
* for literals, scalars and inside the optimizer

Type validation happens per type hierarchy (numeric vs decimal) not type
Ceil/Floor/Round functions return long/int instead of double
ScalarFunction preserves ProcessorDefinition instead of functionId

Original commit: elastic/x-pack-elasticsearch@a703f8b455
2017-09-20 19:32:37 +03:00
Nik Everett b8f5720283 Switch SqlGetIndicesAction after merge
Merging picked up a change that starts to migrate away from Streamable
and we have to handle it.

Original commit: elastic/x-pack-elasticsearch@fa3489e128
2017-09-20 12:29:42 -04:00
Nik Everett 98029ecaf8 Re-add member mistakenly removed
Original commit: elastic/x-pack-elasticsearch@487369b74c
2017-09-20 12:00:34 -04:00
Nik Everett 6ea902f913 SQL: NOCOMMIT cleanup
Removes a few NOCOMMITs that are tracked other places and updates
a few with plans on how to work on them.

Original commit: elastic/x-pack-elasticsearch@8d1cfdf4ee
2017-09-20 11:19:05 -04:00
Nik Everett 1ede8cadf7 Ignore tabs in sql test keystore files (elastic/x-pack-elasticsearch#2544)
These are just tabs in binary data....

Original commit: elastic/x-pack-elasticsearch@bcfa3feec0
2017-09-19 09:32:42 -04:00
Nik Everett 76543491bc SQL: Soften NOCOMMIT to TODO
I'm adding this to the SQL tracker issue. It is more low priority
but we should think about doing it before merging because it is
fairly easy:

Exceptions have many constructors that take a string pattern and
arguments. These kinds of things have mostly been removed in core
because we prefer explicitly building the message at the call site
for clarity. We should probably remove these constructors because
it is easy and another way that we can be more like core quickly.

Original commit: elastic/x-pack-elasticsearch@fa4326c4d1
2017-09-18 18:07:59 -04:00
Nik Everett 08a09582bc Remove two more NOCOMMITS from SQL
They are being tracked in the sql tracker issue as release blockers.

Original commit: elastic/x-pack-elasticsearch@783544dde5
2017-09-18 18:06:33 -04:00
Nik Everett bc03aa6c03 Line up sql serialization methods with core's (elastic/x-pack-elasticsearch#2538)
This renames that `write` and `read` methods in SQL to `writeTo` and
`readFrom` to line up with the names used in core. I don't have a
strong opinion whether or not any name is better than any other but
I figure there isn't a good reason for SQL to be different from the
rest of Elasticsearch.

Original commit: elastic/x-pack-elasticsearch@e5de9a4b81
2017-09-18 16:06:14 -04:00
Nik Everett 204e340397 Drop parameterized type from SQL's exception messages (elastic/x-pack-elasticsearch#2540)
The type parameter wasn't buying us anything.

Original commit: elastic/x-pack-elasticsearch@5005b26c09
2017-09-18 16:06:05 -04:00
Nik Everett 759411571e Remove two NOCOMMITs
* TimeoutInfo - This is now tracked in the SQL tracker github issue
* AbstractProto - Convert to a TODO as we *can* handle it after
release. I've added it to the SQL tracker github issue in a special
section for low priority protocol stuff. Protocol stuff is special
because if we can make the change before release we don't have to
worry about backwards compatibility.

Original commit: elastic/x-pack-elasticsearch@dbef9db5f8
2017-09-18 12:54:40 -04:00
Nik Everett 4994ff86f2 SQL: Enable tests for shared-proto
There used to not be tests in this project. There are now.

Original commit: elastic/x-pack-elasticsearch@81bd25479b
2017-09-14 12:06:47 -04:00
Nik Everett 5d3f5cc4f8 Support scrolling in SQL's CLI (elastic/x-pack-elasticsearch#2494)
* Move CLI to TransportSqlAction
    * Moves REST endpoint from `/_cli` to `/_sql/cli`
    * Removes the special purpose CLI transport action instead
    implements the CLI entirely on the REST layer, delegating
    all SQL stuff to the same action that backs the `/_sql` REST
    API.
    * Reworks "embedded testing mode" to use a `FilterClient` to
    bounce capture the sql transport action and execute in embedded.
    * Switches CLI formatting from consuming the entire response
    to consuming just the first page of the response and returning
    a `cursor` that can be used to read the next page. That read is
    not yet implemented.
    * Switch CLI formatting from the consuming the `RowSetCursor` to
    consuming the `SqlResponse` object.
    * Adds tests for CLI formatting.
* Support next page in the cli
    * Rename cli's CommandRequest/CommandResponse to
    QueryInitRequest/QueryInitResponse to line up with jdbc
    * Implement QueryPageRequest/QueryPageResponse in cli
    * Use `byte[]` to represent the cursor in the cli. Those bytes
    mean something, but only to the server. The only reasonint that
    the client does about them is "if length == 0 then there isn't a
    next page."
    * Pull common code from jdbc's QueryInitRequest, QueryPageRequest,
    QueryInitResponse, and QueryPageResponse into the shared-proto
    project
        * By implication this switches jdbc's QueryPageRequest to using
     the same cursor implementation as the cli

Original commit: elastic/x-pack-elasticsearch@193586f1ee
2017-09-14 10:26:42 -04:00
Igor Motov 6c0de3689f SQL: Provide hex version of toString for SQL's byte array (elastic/x-pack-elasticsearch#2469)
Original commit: elastic/x-pack-elasticsearch@8eadc9af99
2017-09-14 09:16:35 +09:00
Nik Everett 8b6a0daa05 Add tests for SqlGetIndicesAction.Request serialization (elastic/x-pack-elasticsearch#2452)
This request is already serializeable so we may as well add
tests for that so we don't break it.

Original commit: elastic/x-pack-elasticsearch@16cca77ccf
2017-09-11 14:37:57 -04:00
Nik Everett 94de0d8041 Replace exception catching with fancy returns (elastic/x-pack-elasticsearch#2454)
Instead of throwing and catching an exception for invalid
indices this returns *why* they are invalid in a convenient
object form that can be thrown as an exception when the index
is required or the entire index can be ignored when listing
indices.

Original commit: elastic/x-pack-elasticsearch@f45cbce647
2017-09-11 14:27:50 -04:00
Igor Motov 6ef174aa19 SQL: restore support for timezones in the DateTimeProcessor (elastic/x-pack-elasticsearch#2450)
The timezone handling seems to be lost in a merge conflict in elastic/x-pack-elasticsearch@20f41e2

Original commit: elastic/x-pack-elasticsearch@92c0ca3572
2017-09-10 17:44:45 -07:00
Nik Everett 3f8bf7ccc8 Integrate sql's metadata with security (elastic/x-pack-elasticsearch#2446)
This integrates SQL's metadata calls with security by creating
`SqlIndicesAction` and routing all of SQL's metadata calls through
it. Since it *does* know up from which indices it is working against
it can be an `IndicesRequest.Replaceable` and integrate with the
existing security infrastructure for filtering indices.

This request is implemented fairly similarly to the `GetIndexAction`
with the option to read from the master or from a local copy of
cluster state. Currently SQL forces it to run on the local copy
because the request doesn't properly support serialization. I'd
like to implement that in a followup.

Original commit: elastic/x-pack-elasticsearch@15f9512820
2017-09-08 10:59:47 -04:00
Igor Motov be32241263 SQL: Trivial gradle files cleanups (elastic/x-pack-elasticsearch#2447)
Resolved a few NOCOMMITS in Gradle files plus some minor cleanups

Original commit: elastic/x-pack-elasticsearch@b422d693b0
2017-09-08 09:43:18 -04:00
Igor Motov 442e99fb3d SQL: Fix script.max_compilations_per_minute -> script.max_compilations_rate
Original commit: elastic/x-pack-elasticsearch@9c93d6f254
2017-09-07 13:28:47 -04:00
Nik Everett c540e1d583 Drop AbstractLifecycleComponent from PlanExecutor (elastic/x-pack-elasticsearch#2375)
In core we prefer not to extend `AbstractLifecycleComponent` because
the reasons for the tradeoffs that it makes are lost to the sands of
time.

Original commit: elastic/x-pack-elasticsearch@ec1a32bbb3
2017-08-30 17:09:29 -04:00
Nik Everett 7a2722e74a Remove checkstyle suppressions for sql/jdbc* (elastic/x-pack-elasticsearch#2379)
Removes the line length checkstyle suppression for `sql/jdbc*` and
fixes all the checkstyle violations. Removes a few files that had
violations that were not used.

Original commit: elastic/x-pack-elasticsearch@031c2ba8e3
2017-08-30 16:40:29 -04:00
Nik Everett fd13c54cdc Remove some collection utils (elastic/x-pack-elasticsearch#2384)
Core doesn't go in for fancy collection utils in general and just
manipulates the required collections in line. In an effort to keep
SQL "more like the rest of Elasticsearch" I'm starting to remove
SQL's `CollectionUtils`.

Original commit: elastic/x-pack-elasticsearch@878ee181cb
2017-08-30 16:33:56 -04:00
Nik Everett 833ff18181 Fix broke test for ComposeProcessor
It had a bad branch in its mutate function.

Original commit: elastic/x-pack-elasticsearch@d35ec8549c
2017-08-30 16:20:27 -04:00
Nik Everett efc876c8e1 Add an example of including sql from docs
Original commit: elastic/x-pack-elasticsearch@aa4ad288dc
2017-08-29 15:32:44 -04:00
Nik Everett 35f98d1af3 Make DebugSqlSpec abstract
It is used for debugging tests and should be kept abstract or
else it violates the naming conventions. It violates the naming
conventions because we do not wish to run it with the normal build.

Original commit: elastic/x-pack-elasticsearch@ce5810b15a
2017-08-28 09:37:00 -04:00
Nik Everett 972c56dafe Begin migrating SQL's next page (elastic/x-pack-elasticsearch#2271)
Scrolling was only implemented for the `SqlAction` (not jdbc or cli)
and it was implemented by keeping request state on the server. On
principle we try to avoid adding extra state to elasticsearch where
possible because it creates extra points of failure and tends to
have lots of hidden complexity.

This replaces the state on the server with serializing state to the
client. This looks to the user like a "next_page" key with fairly
opaque content. It actually consists of an identifier for the *kind*
of scroll, the scroll id, and a base64 string containing the field
extractors.

Right now this only implements scrolling for `SqlAction`. The plan
is to reuse the same implementation for jdbc and cli in a followup.

This also doesn't implement all of the required serialization.
Specifically it doesn't implement serialization of
`ProcessingHitExtractor` because I haven't implemented serialization
for *any* `ColumnProcessors`.

Original commit: elastic/x-pack-elasticsearch@a8567bc5ec
2017-08-28 08:46:49 -04:00
Costin Leau 928c750585 Add SQL queries with NOT comparison
Original commit: elastic/x-pack-elasticsearch@88c76be0a4
2017-08-26 23:39:33 +03:00
Costin Leau a040f3bb14 More integration tests (BETWEEN) plus minor polishing
Original commit: elastic/x-pack-elasticsearch@1546713d5c
2017-08-26 23:14:20 +03:00
Costin Leau 2493685e5e Add support for IS NOT? NULL
Translated to exists and must_not exists queries

Original commit: elastic/x-pack-elasticsearch@534157ecaa
2017-08-26 01:06:25 +03:00
Nik Everett 29c57bbe0c Handle unshading
Original commit: elastic/x-pack-elasticsearch@5f73cecafb
2017-08-25 17:09:49 -04:00
Nik Everett 337e9f4d6e SQL: Ignore the _default_ type
It is allowed even in single type indices because it isn't real. We should
ignore it entirely.

Original commit: elastic/x-pack-elasticsearch@efc2cf80c8
2017-08-24 17:43:22 -04:00
Nik Everett 65e9889e51 Fix toString
Broken by merging master.

Original commit: elastic/x-pack-elasticsearch@229a92a6be
2017-08-23 12:47:44 -04:00
Nik Everett 1ea3f5175a Fix build
Original commit: elastic/x-pack-elasticsearch@474e5e7a9a
2017-08-23 09:17:39 -04:00
Costin Leau 18dccbc668 Consolidate spec testing
Remove some duplicated methods, add some templating plus logging of
ES resultset (for easier debugging)
Rename debug test for CSV plus add one for Sql spec

Original commit: elastic/x-pack-elasticsearch@d2c46a2ed2
2017-08-22 23:24:01 +03:00
Igor Motov 4c597a2944 SQL: Add support for explicit types in CsvSpecTests (elastic/x-pack-elasticsearch#2306)
Adds support for specifying column types in CSV Spec tests.

Original commit: elastic/x-pack-elasticsearch@4d5622f298
2017-08-18 12:12:06 -04:00
Igor Motov 65ff793a19 SQL: Rename DebugCsvSpec to match test naming convention
Original commit: elastic/x-pack-elasticsearch@7ad2dba5ec
2017-08-17 16:25:11 -04:00
Costin Leau 4429defcb1 Fix hierarchy of Percentile/PercentileRank
Original commit: elastic/x-pack-elasticsearch@b56e65a010
2017-08-17 17:29:32 +03:00
Costin Leau 8605263385 Fix overlapping getters in Function vs AggFunction
Original commit: elastic/x-pack-elasticsearch@f208936b72
2017-08-17 17:29:16 +03:00
Costin Leau 823bea50d0 Count is not a NumericAggregate
Original commit: elastic/x-pack-elasticsearch@7888ca231f
2017-08-17 17:11:14 +03:00
Costin Leau 4907f3bd54 Improve parsing of Limit
Original commit: elastic/x-pack-elasticsearch@7d1231352a
2017-08-17 17:11:04 +03:00
Costin Leau 1471c8ab7f Update constructor for NumericAggregate classes
Original commit: elastic/x-pack-elasticsearch@732742083b
2017-08-17 16:20:44 +03:00
Costin Leau a4bd077913 catch-up to percentile ranks agg change in es master
Original commit: elastic/x-pack-elasticsearch@b7428ab61f
2017-08-16 21:45:19 +03:00
Costin Leau 2c830bdff2 Add support for percentile and percentile ranks aggs
Original commit: elastic/x-pack-elasticsearch@dc443ed465
2017-08-16 12:18:20 +03:00
Igor Motov 4242cc7370 SQL: Tests cleanup (elastic/x-pack-elasticsearch#2252)
Move Transport Request/Response tests to sql server project, remove unused ProtoTests.

Original commit: elastic/x-pack-elasticsearch@2ad0f3bde2
2017-08-14 17:23:57 -04:00
Igor Motov a0de35d801 SQL: Add licensing support (elastic/x-pack-elasticsearch#2213)
Adds a granular licensing support to SQL. JDBC now requires a platinum license, everything else work with any non-expired license.

Original commit: elastic/x-pack-elasticsearch@a30470e2c9
2017-08-10 19:19:09 -04:00
Nik Everett f7d526036e Un-AwaitsFix SQL's jdbc integration tests (elastic/x-pack-elasticsearch#2217)
Most tests worked fine. The datetime tests are broken for some time
zones. The csv tests were broken because they accepted the default
fetch size which looks like it is broken.

Original commit: elastic/x-pack-elasticsearch@e034c2f102
2017-08-09 08:33:14 -04:00
Nik Everett 85937c6e78 Fix SQL's DatabaseMetaData (elastic/x-pack-elasticsearch#2214)
It was failing because it was missing config needed to properly
handle time zones.

Original commit: elastic/x-pack-elasticsearch@1166ea7475
2017-08-08 17:48:03 -04:00
Nik Everett e5973ec356 Un-@AwaitsFix ExplainIT
Original commit: elastic/x-pack-elasticsearch@dde8580afc
2017-08-08 16:09:28 -04:00
Nik Everett b31bc786d3 Remove isValidIndex from SQL's Catalog interface (elastic/x-pack-elasticsearch#2195)
This moves validating that each index contains only a single type
into EsCatalog which removes a race condition and removes a method
from the Catalog interface. Removing methods from the Catalog
interface is nice because the fewer methods that the interface has
the fewer have to be secured.

Original commit: elastic/x-pack-elasticsearch@85cd089e47
2017-08-08 15:42:47 -04:00
Nik Everett 570b66638e Add test for index with two types (elastic/x-pack-elasticsearch#2194)
Adds a test that shows *how* SQL fails to address an index with two types
to the full cluster restart tests. Because we're writing this code
against 7.0 don't actually execute the test, but we will execute it when
we merge to 6.x and it *should* work.

Original commit: elastic/x-pack-elasticsearch@b536e9a142
2017-08-08 13:32:13 -04:00
Igor Motov e09eb41340 SQL: make JdbcAction transport friendly (elastic/x-pack-elasticsearch#2200)
Adds proper serialization to JdbcAction's requests and responses

Original commit: elastic/x-pack-elasticsearch@22d2c0582e
2017-08-08 10:26:13 -04:00
Igor Motov cc41a2daa0 SQL: make CliAction transport friendly (elastic/x-pack-elasticsearch#2198)
Adds proper serialization to CliAction's requests and responses

Original commit: elastic/x-pack-elasticsearch@2d2a15a5ba
2017-08-07 16:15:11 -04:00
Igor Motov 0844ada594 SQL: make SqlAction transport friendly (elastic/x-pack-elasticsearch#2188)
Adds proper serialization to SqlAction's requests and responses

Original commit: elastic/x-pack-elasticsearch@113f69d5b9
2017-08-04 16:25:38 -04:00
Nik Everett f241512e33 SQL support for field level security (elastic/x-pack-elasticsearch#2162)
This adds support for field level security to SQL by creating a new type of flow for securing requests that look like sql requests. `AuthorizationService` verifies that the user can execute the request but doesn't check the indices in the request because they are not yet ready. Instead, it adds a `BiFunction` to the context that can be used to check permissions for an index while servicing the request. This allows requests to cooperatively secure themselves. SQL does this by implementing filtering on top of its `Catalog` abstraction and backing that filtering with security's filters. This minimizes the touch points between security and SQL.

Stuff I'd like to do in followups:

What doesn't work at all still:
1. `SHOW TABLES` is still totally unsecured
2. `DESCRIBE TABLE` is still totally unsecured
3. JDBC's metadata APIs are still totally unsecured

What kind of works but not well:
1. The audit trail doesn't show the index being authorized for SQL.

Original commit: elastic/x-pack-elasticsearch@86f88ba2f5
2017-08-04 15:27:27 -04:00
Nik Everett 4f42de6b1a Fix as many busted tests as I can
Original commit: elastic/x-pack-elasticsearch@5ec24f6818
2017-08-03 17:27:56 -04:00
Nik Everett 5f7bb7587a Silence broken test
Original commit: elastic/x-pack-elasticsearch@9c556a7104
2017-08-03 15:35:02 -04:00
Costin Leau 0675d08f25 Remove support for indices with multiple types
Indices discovery actively ignores indices with more than one type.
However queries made such indices throw an exception (assuming the user
by mistake or not, selects such an index).

Original commit: elastic/x-pack-elasticsearch@16855c7b8f
2017-08-03 22:05:59 +03:00
Costin Leau 38ea150f17 Expand SQL client support
- SSL (tested)
- Basic auth
- Proxying (HTTP and SOCKS)

Original commit: elastic/x-pack-elasticsearch@3abc49897e
2017-08-03 15:36:03 +03:00
Nik Everett 0605802d22 Add a multi-node test to sql (elastic/x-pack-elasticsearch#2136)
SQL relies on being able to fetch information about fields from
the cluster state and it'd be disasterous if that information
wasn't available. This should catch that.

Original commit: elastic/x-pack-elasticsearch@1a62747332
2017-08-02 14:39:25 -04:00
Nik Everett 767a43ca44 Move sql rest test into qa (elastic/x-pack-elasticsearch#2149)
Running the sql rest action test inside the server caused a dependency
loop which was failing the build.

Original commit: elastic/x-pack-elasticsearch@43283671d8
2017-08-01 17:23:07 -04:00
Nik Everett 89071f1807 Remove a dependency cycle in the build (elastic/x-pack-elasticsearch#2122)
This is a hack to remove a dependency cycle I added in elastic/x-pack-elasticsearch#2109. I think
it'd be cleaner to remove the cycle by making sql its own plugin that
doesn't depend on the rest of x-pack-elasticsearch but is still
included within x-pack-elasticsearch. But that is a broader change.

Original commit: elastic/x-pack-elasticsearch@47b7d69d80
2017-08-01 10:29:49 -04:00
Nik Everett fcdf36c379 Fix missing columns to be a 400 error (elastic/x-pack-elasticsearch#2123)
1. We talked about doing 422 but that doesn't work because the netty plugin
maps 422 -> 400 and I didn't think it was worth changing that right now.
2. Missing columns in the `WHERE` clause still cause a 500 response because
we don't resolve the `SELECT` part if there is an error in the `WHERE`
clause.

Original commit: elastic/x-pack-elasticsearch@355d8292e5
2017-07-31 09:44:57 -04:00
Nik Everett 2379c40f58 Add basic integration test for /_sql endpoint (elastic/x-pack-elasticsearch#2109)
Similar to my work in elastic/x-pack-elasticsearch#2106, this adds a basic integration test for the rest sql action. We can add more as we go. Specifically, I'd like to add testing around handling of invalid SQL and a test for timezones, but neither work particularly well yet.

Original commit: elastic/x-pack-elasticsearch@923d941d0d
2017-07-28 12:51:43 -04:00
Nik Everett 38e31cb4f6 Fix compiling sql after merge
Original commit: elastic/x-pack-elasticsearch@c0fc122e81
2017-07-28 12:39:08 -04:00
Nik Everett ec36875872 Sql security take 1 (elastic/x-pack-elasticsearch#2106)
Add some basic security testing/integration.

The good news:
1. Basic security now works. Users without access to an index can't run sql queries against it. Without this change they could.
2. Document level security works! At least so far as I can tell.

The work left to do:
1. Field level security doesn't work properly. I mean, it kind of works in that the field's values don't leak but it just looks like they all have null values.
2. We will need to test scrolling.
3. I've only added tests for the rest sql action. I'll need to add tests for jdbc and the CLI as well.
4. I've only added tests for `SELECT` and have ignored stuff like `DESCRIBE` and `SHOW TABLES`.

Original commit: elastic/x-pack-elasticsearch@b9909bbda0
2017-07-27 17:40:14 -04:00
Nik Everett 2ac45f18fe Remove leftover in build (elastic/x-pack-elasticsearch#2099)
I had added this to help VersionTests.java identify when it was running
in gradle so it could be sure that it got the real Version information
but even running in gradle doesn't give the real version information
because it runs against a build directory instead of a jar. And the jar
is all that has the version information because the version information
is read from the manifest.

Original commit: elastic/x-pack-elasticsearch@3acbd01433
2017-07-27 10:01:09 -04:00
Igor Motov 2c8f96efc7 Sql should run in ci (elastic/x-pack-elasticsearch#2080)
Fixes several minor issue to enable sql branch to run in CI.

Original commit: elastic/x-pack-elasticsearch@b479cff92c
2017-07-25 14:59:50 -04:00
Nik Everett dd29035e66 Use manifest file for version info
This removes a hack I'd left in the build file that hard coded the
hash of jdbc driver. Now we dig the version information out of the
MANIFEST.MF file that is written during the jar process for all
projects in the Elasticsearch build.

Original commit: elastic/x-pack-elasticsearch@fa01cc6fb3
2017-07-25 13:30:35 -04:00
Nik Everett a8a6191428 Handle core's shading and fix some licenses
Original commit: elastic/x-pack-elasticsearch@962dd2c8b4
2017-07-25 09:41:52 -04:00
Costin Leau 04c9427596 Pass jdbc fetch size to the scroller
Original commit: elastic/x-pack-elasticsearch@2309a43980
2017-07-21 00:11:01 +03:00
Costin Leau 76b429bfe2 Introduce SqlSettings for handling per-client configuration
To avoid leaking client information across the entire code-base, client
settings like TimeZone or pagination are stored in
SqlSession>SqlSettings which are available as a ThreadLocal (during
analysis) so that components that need them, can pick them up.

Since ES internally uses Joda, the date/time functionality relies on Joda,
whenever possible to match the behavior.

Original commit: elastic/x-pack-elasticsearch@20f41e2bb3
2017-07-20 19:28:04 +03:00
Costin Leau 8acacc4f7d Add random fetchSize
Original commit: elastic/x-pack-elasticsearch@1f8197b04f
2017-07-18 23:15:40 +03:00