Commit Graph

74 Commits

Author SHA1 Message Date
Mark Vieira e44b8b1e2e
[Backport] Remove dependency substitutions 7.x (#42866)
* Remove unnecessary usage of Gradle dependency substitution rules (#42773)

(cherry picked from commit 12d583dbf6f7d44f00aa365e34fc7e937c3c61f7)
2019-06-04 13:50:23 -07:00
Igor Motov d2f9ccbe18 Geo: Refactor libs/geo parsers (#42549)
Refactors the WKT and GeoJSON parsers from an utility class into an
instantiatable objects. This is a preliminary step in
preparation for moving out coordinate validators from Geometry
constructors. This should allow us to make validators plugable.
2019-05-29 20:07:27 -04:00
Costin Leau d5f04d29c9 SQL: Add support for FROZEN indices (#41558)
Allow querying of FROZEN indices both through dedicated SQL grammar
extension:
> SELECT field FROM FROZEN index
and also through driver configuration parameter, namely:
> index.include.frozen: true/false

Fix #39390
Fix #39377

(cherry picked from commit 2445a933915f420c7f51e8505afa0a7978ce6b0f)
2019-05-23 21:16:16 +03:00
Igor Motov 70ea3cf847
SQL: Add initial geo support (#42031) (#42135)
Adds an initial limited implementations of geo features to SQL. This implementation is based on the [OpenGIS® Implementation Standard for Geographic information - Simple feature access](http://www.opengeospatial.org/standards/sfs), which is the current standard for GIS system implementation. This effort is concentrate on SQL option AKA ISO 19125-2. 

Queries that are supported as a result of this initial implementation

Metadata commands

- `DESCRIBE table`  - returns the correct column types `GEOMETRY` for geo shapes and geo points.
- `SHOW FUNCTIONS` - returns a list that includes supported `ST_` functions
- `SYS TYPES` and `SYS COLUMNS` display correct types `GEO_SHAPE` and `GEO_POINT` for geo shapes and geo points accordingly. 

Returning geoshapes and geopoints from elasticsearch

- `SELECT geom FROM table` - returns the geoshapes and geo_points as libs/geo objects in JDBC or as WKT strings in console.
- `SELECT ST_AsWKT(geom) FROM table;` and `SELECT ST_AsText(geom) FROM table;`- returns the geoshapes ang geopoints in their WKT representation;

Using geopoints to elasticsearch

- The following functions will be supported for geopoints in queries, sorting and aggregations: `ST_GeomFromText`, `ST_X`, `ST_Y`, `ST_Z`, `ST_GeometryType`, and `ST_Distance`. In most cases when used in queries, sorting and aggregations, these function are translated into script. These functions can be used in the SELECT clause for both geopoints and geoshapes. 
- `SELECT * FROM table WHERE ST_Distance(ST_GeomFromText(POINT(1 2), point) < 10;` - returns all records for which `point` is located within 10m from the `POINT(1 2)`. In this case the WHERE clause is translated into a range query.

Limitations:

Geoshapes cannot be used in queries, sorting and aggregations as part of this initial effort. In order to fully take advantage of geoshapes we would need to have access to geoshape doc values, which is coming in #37206. `ST_Z` cannot be used on geopoints in queries, sorting and aggregations since we don't store altitude in geo_point doc values.

Relates to #29872
Backport of #42031
2019-05-14 18:57:12 -05:00
Nhat Nguyen c4cb0507b4 Mute testDriverConfigurationWithSSLInURL
Tracked at #41557
2019-04-25 17:52:57 -04:00
Costin Leau 38f471ae1c SQL: Change schema calls to empty set (#41034)
As empty string has a certain meaning, the JDBC driver returns an empty
set instead for better client compatibility.

Fix #41028

(cherry picked from commit 4cbafa585b7a514eb6c156606dd516324cd3980a)
2019-04-10 13:14:12 +03:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Costin Leau 2ac514b909 SQL: Fix catalog filtering in SYS COLUMNS (#40583)
Properly treat '%' as a wildcard for catalog filtering instead of doing
a straight string match.
Table filtering now considers aliases as well.
Add escaping char for LIKE queries with user defined params
Fix monotony of ORDINAL_POSITION
Add integration test for SYS COLUMNS - currently running only inside
single_node since the cluster name is test dependent.
Add pattern unescaping for index names

Fix #40582

(cherry picked from commit 8e61b77d3f849661b7175544f471119042fe9551)
2019-04-09 18:44:27 +03:00
Costin Leau af874635ba SQL: Prefer resultSets over exceptions in metadata (#40641)
Changed the JDBC metadata to return empty results sets instead of
throwing SQLFeatureNotSupported as it seems a more safer/compatible
approach for consumers.

Fix #40533

(cherry picked from commit ef2d2527c2b5140556fd477e7ff6ea36966684da)
2019-04-08 16:45:34 +03:00
Marios Trivyzas 3844da318f
SQL: Fix deserialisation issue of TimeProcessor (#40776)
TimeProcessor didn't implement `getWriteableName()` so the one from
the parent was used which returned the `NAME` of the parent. This
caused `TimeProcessor` objects to be deserialised into
DateTimeProcessor.

Moreover, added a restriction to run the TIME related integration tests
only in UTC timezone.

Fixes: #40717

(cherry picked from commit cfea348bec20e547df72c415cccd85279accb767)
2019-04-03 16:42:56 +02:00
Marios Trivyzas 899ed2bf81 SQL: Introduce SQL TIME data type (#39802)
Support ANSI SQL's TIME type by introductin a runtime-only
ES SQL time type.

Closes: #38174
(cherry picked from commit 046ccd4cf0a251b2a3ddff6b072ab539a6711900)
2019-04-01 23:57:27 +02:00
krisds 336c92e4d3 [Contradictory type checks](https://lgtm.com/projects/g/elastic/elasticsearch/snapshot/dist-1916470085-1548143539391/files/x-pack/plugin/sql/jdbc/src/main/java/org/elasticsearch/xpack/sql/jdbc/debug/Debug.java#xe223478b74721d35:1) (#38154) 2019-03-29 17:01:14 +01:00
Costin Leau 5485efa2af SQL: Polish behavior of SYS TABLES command (#40535)
SYS TABLES meta command has been improved to better adhere to the ODBC
spec in particular with regards to the handling of enumerations (and
the differences between '%', null and ''(empty string))

Fix #40348

(cherry picked from commit e3070615000228c283d17ce8d182b44f1450a5d5)
2019-03-28 10:08:31 +02:00
Marios Trivyzas 8e049c5f58
SQL: Fix getTime() methods in JDBC (#40484)
Previously, `getTime(colIdx/colLabel)` and
`getObject(colIdx/colLabel, java.sql.Time.class)` methods were computing
the time from a `ZonedDateTime` by applying day in millis modulo on the epoch millis
of the `ZonedDateTime` object. This is wrong as we need to keep the time
related fields at the timezone of the `ZonedDateTime` object and just
set the date info to the epoch date (01/01/1970).

Additionally fixes a testing issue as the original timezone id is converted
to an offset string when parsing the response from the server.
2019-03-27 17:14:08 +01:00
Marios Trivyzas 143db10980
SQL: Fix issue timezone issues with JDBC getDate/getTime (#40360)
Previously, `getDate(int columnIdx)/getDate(String columnLabel)` and
were using legacy`java.util.Calendar` instead of the the `java.time.*`
classes to reset to the start of day. This resulted in different results 
for certain timestamps and timezones when calling
`getDate(col)` vs`getObject(col, java.sql.Date)`

Now only the methods (that must be implemented due to the JDBC spec)
`getDate(int columnIdx, Calendar cal)/getDate(String columnLabel, Calendar cal)`
are still using the `java.util.Calendar` for those conversion.

The same change was applied to
`getTime(int columnIdx)/getTime(String columnLabel)`
and
`getTimestamp(int columnIdx)/getTimestamp(String columnLabel)`

Fixes: #40289

(cherry picked from commit 44560671f18397e0c58e3647732880fcb73a5034)
2019-03-23 17:01:08 +01:00
Costin Leau 558adc0f28 SQL: Add missing handling of IP field in JDBC (#40384)
Fix #40358

(cherry picked from commit ee286fa4893817637c05d72b93b254b36efc0dae)
(cherry picked from commit d2296249499e31bd512390ac3d20bc38009612b3)
2019-03-23 12:58:10 +02:00
Marios Trivyzas e1eb683c51 SQL: Fix issue with getting DATE type in JDBC (#40207)
Previously, calling getDate()/getTime()/getTimestamp() and getObject()
with the corresponding java.sql class on a column of SQL DATE type from
the JDBC result set would throw an Exception.
2019-03-21 01:48:06 +01:00
Costin Leau 076a68007c SQL: Add multi_value_field_leniency inside FieldHitExtractor (#40113)
For cases where fields can have multi values, allow the behavior to be
customized through a dedicated configuration field.
By default this will be enabled on the drivers so that existing datasets
work instead of throwing an exception.
For regular SQL usage, the behavior is false so that the user is aware
of the underlying data.

Fix #39700

(cherry picked from commit 2b351571961f172fd59290ee079126bbd081ceaf)
2019-03-18 14:56:03 +02:00
Marios Trivyzas 4e9657f93f SQL: Fix bug with JDBC timezone setting and DATE type (#39978)
Previously, JDBC's REST call to the server was always sending UTC
instead of the timezone passed through connection string/properties.

Moreover the conversion to java.sql.Date was problematic as a
calculation on the epoch millis was used to set the time to 00:00:00.000
and the timezone info was lost. This caused the resulting java.sql.Date
object which is always using the JVM's timezone (no matter what timezone
setting is used in the connection string/properties) to be wrongly created.

Fixes: #39915
2019-03-14 13:41:53 +01:00
Andrei Stefan 4deb69e9e4 SQL: introduce the columnar option for REST requests (#39287)
* Add "columnar" option for REST requests (but be lenient for non-"plain"
modes) for json, yaml, smile and cbor formats.
* Updated documentation

(cherry picked from commit 5b7e0de237fb514d14a61a347bc669d4b4adbe56)
2019-02-27 09:37:28 +02:00
Andrei Stefan c1018db404 SQL: enforce JDBC driver - ES server version parity (#38972)
(cherry picked from commit 822a21f29491f295b22dacd04b747781a69ffa61)
2019-02-20 11:29:02 +02:00
Andrei Stefan 92206c8567 Added "validate.properties" property to JDBC's list of allowed properties. (#39050)
This defaults to "true" (current behavior) and will throw an exception
if there is a property that cannot be recognized. If "false", it will
ignore anything unrecognizable.

(cherry picked from commit 38fbf9792bcf4fe66bb3f17589e5fe6d29748d07)
2019-02-20 11:29:01 +02:00
Andrei Stefan 7d78f4641b SQL: fall back to using the field name for column label (#38842)
(cherry picked from commit 0567bf24957be477e7649cff94872b0e7dc4d284)
2019-02-14 14:10:59 +02:00
Marios Trivyzas af8a444caa
SQL: Replace joda with java time (#38437)
Replace remaining usages of joda classes with java time.

Fixes: #37703
2019-02-08 22:58:07 +02:00
Andrei Stefan 22d3290078
SQL: Added SSL configuration options tests (#37875)
* Added SSL configuration options tests
Removed the allow.self.signed option from the documentation since we allow
by default self signed certificates as well.

* Added more tests
2019-01-31 10:52:49 +02:00
Marios Trivyzas f707fa9e0a
SQL: Introduce SQL DATE data type (#37693)
* SQL: Introduce SQL DATE data type

Support ANSI SQL's DATE type by introducing a runtime-only
ES SQL date type.

Closes: #37340
2019-01-24 13:41:58 +02:00
Andrei Stefan 7507af29fa
SQL: Return Intervals in SQL format for CLI (#37602)
* Add separate CLI Mode
* Use the correct Mode for cursor close requests
* Renamed CliFormatter and have different formatting behavior for CLI and "text" format.
2019-01-22 14:55:28 +02:00
Marios Trivyzas 1686c32ba9
SQL: Rename SQL type DATE to DATETIME (#37395)
* SQL: Rename SQL data type DATE to DATETIME

SQL data type DATE has only the date part (e.g.: 2019-01-14)
without any time information. Previously the SQL type DATE was
referring to the ES DATE which contains also the time part along
with TZ information. To conform with SQL data types the data type
`DATE` is renamed to `DATETIME`, since it includes also the time,
as a new runtime SQL `DATE` data type will be introduced down the road,
which only contains the date part and meets the SQL standard.

Closes: #36440

* Address comments
2019-01-17 10:17:58 +02:00
Costin Leau 023bb2f1e4
SQL: Remove slightly used meta commands (#37506)
Remove SYS CATALOGS and SYS TABLE TYPES as they are a subset of SYS
TABLES (and thus somewhat redundant) and used only by JDBC.

Close #37409
2019-01-16 12:36:35 +02:00
Marios Trivyzas 5f2fbedd8c
SQL: Replace String.format() with LoggerMessageFormat.format() (#37216)
Fixes: #36532
2019-01-08 23:56:00 +02:00
Andrei Stefan d31eaf7313
SQL: protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests (#36800)
* Change the way the protocol returns date fields from Long values in case
of JDBC/ODBC, to ISO 8601 with millis String.
2018-12-19 16:36:16 +02:00
Alpar Torok e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task (#36311)
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Costin Leau b376edf269
SQL: Move internals from TimeZone to ZoneId (#36651)
As the internals have moved to java.time, the usage of TimeZone itself
 should be minimized as it creates issues when being converted to ZoneId
Protocol wise the two are mostly identical so consumer should not see
 any difference.
Note that terminology wise, inside the docs, the public API and inside
 the protocol timeZone will continue to be used as it's more widely
 understood as oppose to zoneId which is an implementation detail
 specific to the JVM

Fix #36535
2018-12-17 13:55:44 +02:00
Costin Leau 6ee6bb55e2
SQL: Introduce HISTOGRAM grouping function (#36510)
Introduce Histogram grouping function for bucketing/grouping data based
 on a given range. Both date and numeric histograms are supported using
 the appropriate range declaration (numbers vs intervals).

SELECT HISTOGRAM(number, 50) AS h FROM index GROUP BY h
SELECT HISTOGRAM(date, INTERVAL 1 YEAR) AS h FROM index GROUP BY h

In addition add multiply operator for Intervals
Add docs for intervals and histogram

Fix #36509
2018-12-14 18:20:37 +02:00
Ryan Ernst 95d9cefea7
Build: Fix jdbc jar pom to not include deps (#36036)
This commit adds back the exclusion of dependencies from the pom file
for the jdbc jar, which was accidentally lost in #32014
2018-11-29 09:39:17 -08:00
Andrei Stefan aabff7318e
SQL: DATABASE() and USER() system functions (#35946) 2018-11-28 15:20:04 +02:00
Costin Leau 95346926ef
SQL: Lock down JDBC driver (#35798)
Move classes under the same package to avoid internal classes being
 exposed to the outside. Remove public visibility outside 3 classes:
 EsDriver, EsDataSource and EsTypes.
The driver only has one package, namely org.elasticsearch.xpack.sql.jdbc
Use Es prefix for classes to ease name conflict and indicate their
 destination

Fix #35437
2018-11-28 12:48:26 +02:00
Ryan Ernst 10447dd962
Build: Fix jdbc jar to include deps (#35602)
This commit adds back bundling of all deps of the sql jdbc jar. This was
lost in a refactoring of how the shadow plugin is handled for the entire
elasticsearch project.
2018-11-26 16:18:35 -08:00
Andrei Stefan 00e6fec718
SQL: XPack FeatureSet functionality (#35725)
* Introduced "client.id" parameter for REST requests
* Bug that made the Verifier run twice, fixed in the Analyzer
* Single node IT and unit testing
2018-11-26 07:11:43 +02:00
Costin Leau f0a3d32602
SQL: Introduce INTERVAL support (#35521)
Introduce INTERVAL as a DataType
Add INTERVAL to the grammar which supports the standard SQL declaration
 (without precision):
 > INTERVAL '1 23:45:01.123456789' DAY TO SECOND
 but also number for single unit intervals:
 > INTERVAL 1 YEAR
 as well as the plurals of the units:
 > INTERVAL 2 YEARS
Interval are internally supported as just another Literal being backed
 by java.time.Period and java.time.Duration
Move JDBC away from JDBCType enum to SQLType interface
Refactor DataType by moving it into server core and adding dedicated (and
 much simpler) JDBC driver type
Improve internal JDBC conversion by normalizing on the DataType
Rename JDBC columnInfo to JdbcColumnInfo to differentiate between it and
 the SQL ColumnInfo

Fix #29990
2018-11-21 15:45:59 +02:00
Alpar Torok 8a85b2eada
Remove build qualifier from server's Version (#35172)
With this change, `Version` no longer carries information about the qualifier,
we still need a way to show the "display version" that does have both
qualifier and snapshot. This is now stored  by the build and red from `META-INF`.
2018-11-07 14:01:05 +02:00
Costin Leau 7570d69254
SQL: Introduce support for IP fields (#34758)
IP fields are recognized and can be used through-out the query

Close #32499
2018-10-24 22:03:11 +03:00
Andrei Stefan 91434f7721
SQL: the SSL default configuration shouldn't override the https protocol if used (#34635)
* The default SSL option shouldn't override the https protocol if specified. Fixes https://github.com/elastic/elasticsearch/issues/33817
2018-10-23 11:04:16 +03:00
Nik Everett 540dfcf23e
JDBC: Fix artifactId in pom (#34478)
We're publishing jdbc into our maven repo as though its artifactId is
`x-pack-sql-jdbc` but the pom listed the artifactId as `jdbc`. This
fixes the pom to line up with where we're publishing the artifact.

Closes #34399
2018-10-16 11:02:10 -04:00
Andrei Stefan d7a94fb6aa
SQL: Functions enhancements (OCTET_LENGTH function, order functions alphabetically, RANDOM function docs) (#34101)
* New OCTET_LENGTH function
* Changed the way the FunctionRegistry stores functions, considering the alphabetic ordering by name
* Added documentation for the RANDOM function
2018-10-09 00:20:18 +03:00
Kazuhiro Sera d45fe43a68 Fix a variety of typos and misspelled words (#32792) 2018-10-03 18:11:38 +01:00
Christoph Büscher ba3ceeaccf
Clean up "unused variable" warnings (#31876)
This change cleans up "unused variable" warnings. There are several cases were we 
most likely want to suppress the warnings (especially in the client documentation test
where the snippets contain many unused variables). In a lot of cases the unused
variables can just be deleted though.
2018-09-26 14:09:32 +02:00
Costin Leau 17605bf1c6
SQL: Move away internally from JDBCType to SQLType (#33913)
Replace JDBCType enum with SQLType interface for extensibility

Fix #33094
2018-09-22 11:03:38 +03:00
Andrei Stefan 6fb7e49b22
SQL: TRUNCATE and ROUND functions (#33779)
* Added TRUNCATE function, modified ROUND to accept two parameters instead of one. Made the second parameter optional for both functions.
* Added documentation for both functions.
2018-09-20 15:29:53 +03:00
Costin Leau 736053c658
SQL: Return functions in JDBC driver metadata (#33672)
Update JDBC database metadata to advertised the supported functions
Add aliases to some date/time functions to match the ODBC spec

Fix #33671
2018-09-14 09:05:34 +03:00