Commit Graph

128 Commits

Author SHA1 Message Date
Costin Leau 38dc005b8d SQL: Update JDBC class name in client screenshots
(cherry picked from commit 79a3b93deb858961ecbe4516ed9d8f209016666f)
2019-03-27 12:21:53 +02:00
Costin Leau 61f49af497 SQL: Spec tests now use classpath discovery ()
To avoid having to specify each spec by hand (which can miss specs to be
added), the test infrastructure now performs classpath discovery so that
each spec added, is automatically considered.

Relates 

(cherry picked from commit d0f60b4425c731509aa8ca765d55f563f866ef90)
2019-03-25 15:22:52 +02:00
Costin Leau 496070fda6 SQL: CAST supports both SQL and ES types ()
Extend CAST to support all data types notations (whether SQL or ES
specific)

Fix 

(cherry picked from commit eb2ee8a344da946920598839a5db76c8bb9bc3fe)
2019-03-22 23:55:51 +02:00
Costin Leau 970cd57368 DOC: polish client docs
(cherry picked from commit 72e413df26791b90475ce2f22c31d29e79bd52b9)
2019-03-22 19:43:56 +02:00
Costin Leau 980ee14f57 DOC: Expand section on ORDER BY aggs ()
(cherry picked from commit 99d2f6fc9864ab972259ef5692129ab49e4a7ab8)
2019-03-22 10:04:52 +02:00
Costin Leau 076a68007c SQL: Add multi_value_field_leniency inside FieldHitExtractor ()
For cases where fields can have multi values, allow the behavior to be
customized through a dedicated configuration field.
By default this will be enabled on the drivers so that existing datasets
work instead of throwing an exception.
For regular SQL usage, the behavior is false so that the user is aware
of the underlying data.

Fix 

(cherry picked from commit 2b351571961f172fd59290ee079126bbd081ceaf)
2019-03-18 14:56:03 +02:00
Lisa Cawley c92476f591 [DOCS] Replaces CCS terms with attributes () 2019-03-15 07:57:51 -07:00
Costin Leau 3960374a6f SQL: Introduce MAD (MedianAbsoluteDeviation) aggregation ()
Add Median Absolute Deviation aggregation

Fix 

(cherry picked from commit 4f09613942a9249d06c74da64ad7e6f362e97f56)
2019-03-15 11:45:15 +02:00
Andrei Stefan ba44f28340 SQL: ignore UNSUPPORTED fields for JDBC and ODBC modes in 'SYS COLUMNS' ()
* SYS COLUMNS will skip UNSUPPORTED field types in ODBC and JDBC, as well.
NESTED and OBJECT types were already skipped in ODBC mode, now they are
skipped in JDBC mode, as well.

(cherry picked from commit 9e0df64b2d36c9069dfa506570468f0522c86417)
2019-03-01 15:26:31 +02:00
Andrei Stefan 542e2c55f6 SQL: change the default precision for CURRENT_TIMESTAMP function ()
(cherry picked from commit dbb93310b083226c96e4bde3eef0079eb01cbca9)
2019-02-27 09:49:42 +02:00
Andrei Stefan 4deb69e9e4 SQL: introduce the columnar option for REST requests ()
* Add "columnar" option for REST requests (but be lenient for non-"plain"
modes) for json, yaml, smile and cbor formats.
* Updated documentation

(cherry picked from commit 5b7e0de237fb514d14a61a347bc669d4b4adbe56)
2019-02-27 09:37:28 +02:00
Andrei Stefan 92206c8567 Added "validate.properties" property to JDBC's list of allowed properties. ()
This defaults to "true" (current behavior) and will throw an exception
if there is a property that cannot be recognized. If "false", it will
ignore anything unrecognizable.

(cherry picked from commit 38fbf9792bcf4fe66bb3f17589e5fe6d29748d07)
2019-02-20 11:29:01 +02:00
Costin Leau c5dce42667 SQL: doc polishing 2019-02-15 22:14:36 +02:00
Costin Leau 79bc6aba79 SQL: Polish the rest chapter ()
Organize the text a bit and add tip on triple quotes in Kibana Console
2019-02-15 22:14:36 +02:00
Costin Leau 9357c17288 SQL: Doc on syntax (identifiers in particular) ()
Add section on syntax, identifiers and literals and on single vs double quotes.

(cherry picked from commit aafdb598082e451f36294bd174d0887a276d8c7f)
2019-02-15 15:24:33 +02:00
taku333 8bb2a2a405 SQL: change JDBC setup URL in the documentation ()
(cherry picked from commit 103786ea27da72b2fccd3cf511b3143dae0fc530)
2019-02-14 17:12:08 +02:00
Marios Trivyzas 032bcf99d6
SQL: Implement `::` cast operator ()
`<expression>::<dataType>` is a simplified altenative syntax to
`CAST(<expression> AS <dataType> which exists in PostgreSQL and
provides an improved user experience and possibly more compact
SQL queries.

Fixes: 
2019-02-12 16:54:14 +02:00
Costin Leau 5b112b1d9d SQL: remove beta marker from documentation ()
(cherry picked from commit fb6e7a30c9eed1e8b83496aaf1efe7e2288f9dd8)
2019-02-10 00:09:58 +02:00
Marios Trivyzas c9701be1e8
SQL: Implement CURRENT_DATE ()
Since DATE data type is now available, this implements the
`CURRENT_DATE/CURRENT_DATE()/TODAY()` similar to `CURRENT_TIMESTAMP`.

Closes: 
2019-02-05 18:15:26 +02:00
Costin Leau 783c9ed372
SQL: Allow sorting of groups by aggregates ()
Introduce client-side sorting of groups based on aggregate
functions. To allow this, the Analyzer has been extended to push down
to underlying Aggregate, aggregate function and the Querier has been
extended to identify the case and consume the results in order and sort
them based on the given columns.
The underlying QueryContainer has been slightly modified to allow a view
of the underlying values being extracted as the columns used for sorting
might not be requested by the user.

The PR also adds minor tweaks, mainly related to tree output.

Close 
2019-02-02 01:38:25 +02:00
Marios Trivyzas 19dccf8f3e
SQL: [Docs] Add limitation for aggregate functions on scalars ()
Currently aggregate functions can operate only directly on fields.
They cannot be used on top of scalar functions as painless scripting
is currently not supported.
2019-02-01 16:13:51 +02:00
Marios Trivyzas 4710a7472f
SQL: Implement FIRST/LAST aggregate functions ()
FIRST and LAST can be used with one argument and work similarly to MIN
and MAX but they are implemented using a Top Hits aggregation and
therefore can also operate on keyword fields. When a second argument is
provided then they return the first/last value of the first arg when its
values are ordered ascending/descending (respectively) by the values of
the second argument. Currently because of the usage of a Top Hits
aggregation FIRST and LAST cannot be used in the HAVING clause of a
GROUP BY query to filter on the results of the aggregation.

Closes: 
2019-01-31 16:33:05 +02:00
Andrei Stefan 22d3290078
SQL: Added SSL configuration options tests ()
* Added SSL configuration options tests
Removed the allow.self.signed option from the documentation since we allow
by default self signed certificates as well.

* Added more tests
2019-01-31 10:52:49 +02:00
Adrien Grand c8af0f4bfa
Use mappings to format doc-value fields by default. ()
Doc-value fields now return a value that is based on the mappings rather than
the script implementation by default.

This deprecates the special `use_field_mapping` docvalue format which was added
in  only to ease the transition to 7.x and it is not necessary anymore in
7.0.
2019-01-30 10:31:51 +01:00
Marios Trivyzas f707fa9e0a
SQL: Introduce SQL DATE data type ()
* SQL: Introduce SQL DATE data type

Support ANSI SQL's DATE type by introducing a runtime-only
ES SQL date type.

Closes: 
2019-01-24 13:41:58 +02:00
Marios Trivyzas 1686c32ba9
SQL: Rename SQL type DATE to DATETIME ()
* SQL: Rename SQL data type DATE to DATETIME

SQL data type DATE has only the date part (e.g.: 2019-01-14)
without any time information. Previously the SQL type DATE was
referring to the ES DATE which contains also the time part along
with TZ information. To conform with SQL data types the data type
`DATE` is renamed to `DATETIME`, since it includes also the time,
as a new runtime SQL `DATE` data type will be introduced down the road,
which only contains the date part and meets the SQL standard.

Closes: 

* Address comments
2019-01-17 10:17:58 +02:00
Marios Trivyzas 8932750efb
SQL: [Docs] Add an ES-SQL column for data types ()
In order to distinguish the ES-SQL type from the standard SQL type
add a new ES-SQL column that will make clear this distingstion,
e.g.: datetime vs TIMSTAMP

Fixes: 
2019-01-16 16:16:21 +02:00
Andrei Stefan 4a92de214a
SQL: Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) ()
* provide overriden `hashCode` and toString methods to account for `DISTINCT`
* change the analyzer for scenarios where `COUNT <field_name>` and `COUNT DISTINCT` have different paths
* defined a new `filter` aggregation encapsulating an `exists` query to filter out null or missing values
2019-01-10 09:51:51 +02:00
Josh Soref edb48321ba [DOCS] Various spelling corrections () 2019-01-07 14:44:12 +01:00
Andrei Stefan 39a072389c
SQL: add sub-selects to the Limitations page () 2019-01-07 10:08:51 +02:00
Marios Trivyzas 33137907cf
SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg ()
Enhance error message for the case that the 2nd argument of PERCENTILE
and PERCENTILE_RANK is not a foldable, as it doesn't make sense to have
a dynamic value coming from a field.

Fixes: 
2019-01-03 13:55:09 +02:00
Andrei Stefan 9c283aed0a
SQL: Move the SQL ODBC client applications pages from the ODBC section into the general ES SQL Client Applications page () 2018-12-31 14:07:39 +02:00
Andrei Stefan 0b1cb5300e
Move ODBC documentation to `elasticsearch` repository () 2018-12-27 11:55:20 +02:00
Andrei Stefan 09fa827adc
SQL: documentation improvements and updates ()
* Added Limitations page
* Made the aggregations page follow the common template for functions
* Modified all tables to have the first row's cells content centered
* Polishing in other various sections
2018-12-21 23:25:54 +02:00
Costin Leau ac032a0b9d
SQL: Fix bug regarding histograms usage in scripting ()
Allow scripts to correctly reference grouping functions
Fix bug in translation of date/time functions mixed with histograms.
Enhance Verifier to prevent histograms being nested inside other
 functions inside GROUP BY (as it implies double grouping)
Extend Histogram docs
2018-12-20 23:11:56 +02:00
Costin Leau 6ee6bb55e2
SQL: Introduce HISTOGRAM grouping function ()
Introduce Histogram grouping function for bucketing/grouping data based
 on a given range. Both date and numeric histograms are supported using
 the appropriate range declaration (numbers vs intervals).

SELECT HISTOGRAM(number, 50) AS h FROM index GROUP BY h
SELECT HISTOGRAM(date, INTERVAL 1 YEAR) AS h FROM index GROUP BY h

In addition add multiply operator for Intervals
Add docs for intervals and histogram

Fix 
2018-12-14 18:20:37 +02:00
Costin Leau f502ca53d4
SQL: Introduce NOW/CURRENT_TIMESTAMP function ()
Add CURRENT_TIMESTAMP as keyword as well function alongside NOW() 
 These return the current date/time for the  given query, computed when
 the statement reaches the server. For completeness, CURRENT_TIMESTAMP
 also accepts precision as an optional parameter.

Fix 
2018-12-14 15:42:36 +02:00
Andrei Stefan de373060fb
SQL: non ISO 8601 versions of DAY_OF_WEEK and WEEK_OF_YEAR functions ()
* Renamed DAY_OF_WEEK and WEEK_OF_YEAR functions to their ISO version and
added the same functions with different functionality.
* Rewritten the datetime functions documentation to follow the format of the other
functions documentation pages.
2018-12-12 02:29:02 +02:00
Marios Trivyzas 031d0566b0
SQL: [docs] Add `beta[]` markup to all SQL pages ()
Closes: 
2018-12-11 13:25:29 +02:00
Andrei Stefan aabff7318e
SQL: DATABASE() and USER() system functions () 2018-11-28 15:20:04 +02:00
Costin Leau 95346926ef
SQL: Lock down JDBC driver ()
Move classes under the same package to avoid internal classes being
 exposed to the outside. Remove public visibility outside 3 classes:
 EsDriver, EsDataSource and EsTypes.
The driver only has one package, namely org.elasticsearch.xpack.sql.jdbc
Use Es prefix for classes to ease name conflict and indicate their
 destination

Fix 
2018-11-28 12:48:26 +02:00
Jason Tedor c42d9d91c9
Deprecate X-Pack centric SQL endpoints ()
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-27 22:16:21 -05:00
Marios Trivyzas 3f7cae3f0d
SQL: Implement GREATEST and LEAST functions ()
Add GREATEST(expr1, expr2, ... exprN) and LEAST(expr1, expr2, exprN)
functions which are in the family of CONDITIONAL functions.

Implementation follows PostgreSQL behaviour, so the functions return
`NULL` when all of their arguments evaluate to `NULL`.

Renamed `CoalescePipe` and `CoalesceProcessor` to `ConditionalPipe` and
`ConditionalProcessor` respectively, to be able to reuse them for
`Greatest` and `Least` evaluations. To achieve that `ConditionalOperation`
has been added to differentiate between the functionalities at execution
time.

Closes: 
2018-11-26 18:21:36 +01:00
Marios Trivyzas b078e2970c
SQL: Implement null safe equality operator `<=>` ()
This operator handles nulls in different way than the normal `=`.
If one of the operants is `null` and the other not it returns `false`.
If both operants are `null` it returns `true`. Therefore in contrary to
`=`, which returns `null` if at least one of the operants is `null`, this one
never returns `null` as a result.

Closes: 
2018-11-26 14:02:02 +01:00
Marios Trivyzas 410f570d5f
SQL: Implement NULLIF(expr1, expr2) function ()
NULLIF returns null if the 2 expressions are equal or the
expr1 otherwise.

Closes: 
2018-11-23 22:19:27 +01:00
Marios Trivyzas 92acf47c16
SQL: Implement NVL(expr1, expr2) ()
Add NVL as alias to IFNULL as they have the same
behaviour. Add basic tests and docs.

Closes: 
2018-11-22 11:41:00 +01:00
Marios Trivyzas d95d885bae
SQL: Implement ISNULL(expr1, expr2) ()
Add ISNULL as an alias of IFNULL as they have the
same behaviour. Add basic test and docs.

Closes: 
2018-11-21 23:15:10 +01:00
Marios Trivyzas e179bd393d
SQL: Implement IFNULL variant of COALESCE ()
IFNULL is a MySQL variant (also used in other DBs) which
takes only 2 arguments and returns the first one that is not null.

Closes: 
2018-11-21 17:07:07 +01:00
Marios Trivyzas b1818dbdce
SQL: [docs] Add documentation for COALESCE ()
Follows: 
2018-11-21 01:43:05 +01:00
Deb Adair 91a38b8fe0 [DOCS] Fixed bad cross doc link. 2018-11-14 16:09:18 -08:00
debadair 38d6bb27f5
[DOCS] Added link to standalone ODBC driver doc. () 2018-11-14 15:23:28 -08:00
Thiago Souza 98e075227b [Doc] Correct jdbc url example ()
Adds a missing `?` to separate the url parameters.
2018-11-14 17:30:23 +01:00
Nik Everett 0f9a7b3344 Docs: Fix build after moving SQL QA
I moved SQL's QA in  to make it easier to run all of SQL tests in
one go but forgot to update the docs build. This fixes that.
2018-10-30 15:47:25 -04:00
Costin Leau ca6808e55d
SQL: Support pattern against compatible indices ()
Extend querying support on multiple indices from being strictly
identical to being just compatible.
Use FieldCapabilities API (extended through ) for mapping merging.

Close  
2018-10-23 17:07:51 +03:00
Marios Trivyzas 4a8386f271
SQL: Implement IN(value1, value2, ...) expression. ()
Implement the functionality to translate the
`field IN (value1, value2,...)` expressions to proper Lucene queries
or painless script or local processors depending on the use case.

The `IN` expression can be used in SELECT, WHERE and HAVING clauses.

Closes: 
2018-10-23 14:28:23 +02:00
Marios Trivyzas e9e140790a
SQL: Implement `CONVERT`, an alternative to `CAST` ()
`CONVERT` works exactly like cast with slightly different syntax:
`CONVERT(<value>, <data_type)` as opposed to `CAST(<value> AS <data_type>)`

Moreover it support format of the MS-SQL data types `SQL_<type>`,
e.g.: `SQL_INTEGER`

Closes: 
2018-10-23 11:21:15 +02:00
Costin Leau 52104aac27
SQL: Introduce support for NULL values ()
Make SQL aware of missing and/or unmapped fields treating them as NULL
Make _all_ functions and operators null-safe aware, including when used
in filtering or sorting contexts
Add missing and null-safe doc value extractor
Modify dataset to have null fields spread around (in groups of 10)
Enforce missing last and unmapped_type inside sorting
Consolidate Predicate templating and declaration
Add support for Like/RLike in scripting
Generalize NULLS LAST/FIRST
Introduce early schema declaration for CSV spec tests: to keep the doc
snippets in place (introduce schema:: prefix for declaration)
upfront.

Fix 
2018-10-19 16:44:33 +03:00
Andrei Stefan d7a94fb6aa
SQL: Functions enhancements (OCTET_LENGTH function, order functions alphabetically, RANDOM function docs) ()
* New OCTET_LENGTH function
* Changed the way the FunctionRegistry stores functions, considering the alphabetic ordering by name
* Added documentation for the RANDOM function
2018-10-09 00:20:18 +03:00
Kazuhiro Sera d45fe43a68 Fix a variety of typos and misspelled words () 2018-10-03 18:11:38 +01:00
Marios Trivyzas a8a2a83d45
SQL: Fix grammar for `*` in arithm expressions ()
Previously, parsing an arithmetic expression with `*` and no spaces,
e.g.: `2*i` threw a parsing exception as the grammar rule for
tableIdentifier was clashing with the rule for arithmetic operator `*`.

This issue comes already in the lexer and the left part of the
expression (in our example `2*`) was recognised as a
TABLE_IDENTIFIER token.

The solution adopted is to allow the `*` wildcard in the table name
only if it's surrounded with double quotes, e.g.: `"my*index"`

Closes: 
2018-10-02 11:47:42 +03:00
Lisa Cawley 68b42f7d89
[DOCS] Moves reserved keywords under SQL section () 2018-09-28 10:17:41 -07:00
Lisa Cawley 949e4e9d1a
[DOCS] Synchronizes captialization in top-level titles () 2018-09-27 08:36:18 -07:00
Andrei Stefan 0fae6d39f5
SQL: functions docs update ()
* Changed the format of the String functions documentation page.
* Adopted the same format for Math functions, but completely changed the examples.
* Added missing documentation for Math functions.
2018-09-25 02:42:18 +03:00
Andrei Stefan 6fb7e49b22
SQL: TRUNCATE and ROUND functions ()
* Added TRUNCATE function, modified ROUND to accept two parameters instead of one. Made the second parameter optional for both functions.
* Added documentation for both functions.
2018-09-20 15:29:53 +03:00
Costin Leau 32a22ca00e DOC: improved wording in SQL client app section 2018-09-13 22:07:23 +03:00
Costin Leau a192785fc8 DOC: Add SQL section on client applications
Add setup instructions for a number of GUI SQL applications
2018-09-13 15:44:52 +03:00
Costin Leau 443f9caddd DOC: Enhance SQL Functions documentation
Split function section into multiple chapters
Add String functions
Add (small) section on Conversion/Cast functions
Add missing aggregation functions
Enable documentation testing (was disabled by accident). While at it,
fix failing tests
Improve spec tests to allow multi-line queries (useful for docs)
Add ability to ignore a spec test (name should end with -Ignore)
2018-09-06 18:09:53 +03:00
lcawl c5109a54ee [DOCS] Revert fix for broken link 2018-09-04 09:26:28 -07:00
Costin Leau 43f80fa82b DOCS: Fix anchor and example typos 2018-09-04 19:06:44 +03:00
lcawl 303ae25a6a [DOCS] Fixes broken link 2018-09-04 09:05:30 -07:00
Costin Leau 17c7f99343
SQL: Show/desc commands now support table ids ()
Extend SHOW TABLES, DESCRIBE and SHOW COLUMNS to support table
identifiers not just SQL LIKE pattern.
This allows both Elasticsearch-style multi-index patterns and SQL LIKE.
To disambiguate between the two (as the " vs ' can be easy to miss),
the grammar now requires LIKE keyword as a prefix for all LIKE-like
patterns.

Also added some docs comparing the two types of patterns.

Fix 
2018-09-04 16:54:10 +03:00
Andrei Stefan 20e1b5e2fe
Different handling for security specific errors in the CLI. Fix for https://github.com/elastic/elasticsearch/issues/33230 () 2018-08-31 13:08:32 +03:00
lipsill b7c0d2830a [Docs] Remove repeating words () 2018-08-28 13:16:43 +02:00
Dominik 525cda0331 Minor spelling and grammar fix () 2018-08-28 12:50:19 +02:00
Costin Leau 9ffb26ab02
SQL: Remove restriction for single column grouping ()
For historical reasons SQL restricts GROUP BY to only one field.
This commit removes the restriction and improves the test suite with
multi group by tests.

Close 
2018-07-06 20:55:27 +03:00
Costin Leau 093ea037b4 [DOCS] Typos 2018-07-03 17:19:48 +03:00
Costin Leau de9e56aa01
DOC: Add examples to the SQL docs ()
Significantly improve the example snippets in the documentation.
The examples are part of the test suite and checked nightly.
To help readability, the existing dataset was extended (test_emp renamed
to emp plus library).
Improve output of JDBC tests to be consistent with the CLI
Add lenient flag to JDBC asserts to allow type widening (a long is
equivalent to a integer as long as the value is the same).
2018-07-03 16:56:31 +03:00
Lisa Cawley 638b9fd88c
[DOCS] Move sql to docs () 2018-06-22 15:40:25 -07:00