Commit Graph

359 Commits

Author SHA1 Message Date
Marios Trivyzas b0cb7bf229 SQL: Fix issue with GROUP BY YEAR() ()
Grouping By YEAR() is translated to a histogram aggregation, but
previously if there was a scalar function invloved (e.g.:
`YEAR(date + INTERVAL 2 YEARS)`), there was no proper script created
and the histogram was applied on a field with name: `date + INTERVAL 2 YEARS`
which doesn't make sense, and resulted in null result.

Check the underlying field of YEAR() and if it's a function call
`asScript()` to properly get the painless script on which the histogram
is applied.

Fixes: 
(cherry picked from commit 93c37abc943d00d3a14ba08435d118a6d48874c7)
2019-11-26 14:11:11 +01:00
Marios Trivyzas 3c69d4d0bd
SQL: Add TRUNC alias for TRUNCATE ()
Add TRUNC as alias to already implemented TRUNCATE
numeric function which is the flavour supported by
Oracle and PostgreSQL.

Relates to: 

(cherry picked from commit f2aa7f0779bc5cce40cc0c1f5e5cf1a5bb7d84f0)
2019-11-26 12:32:54 +01:00
Marios Trivyzas 5d306ae3b2
SQL: Fix issue with CASE/IIF pre-calculating results ()
Previously, CaseProcessor was pre-calculating (called `process()`)
on all the building elements of a CASE/IIF expression, not only the
conditions involved but also the results, as well as the final else result.
In case one of those results had an erroneous calculation
(e.g.: division by zero) this was executed and resulted in
an Exception to be thrown, even if this result was not used because of
the condition guarding it. e.g.:

```
SELECT CASE myField1 = 0 THEN NULL ELSE myField2 / myField1 END
FROM test;
```

Fixes: 
(cherry picked from commit dbd169afc98686cae1bc72024fad0ca32b272efd)
2019-11-26 10:48:07 +01:00
Marios Trivyzas 0c4491964b SQL: Fix issue with folding of CASE/IIF ()
Add extra checks to prevent ConstantFolding rule to try to fold
the CASE/IIF functions early before the SimplifyCase rule gets applied.

Fixes: 

(cherry picked from commit f35c9725350e35985d8dd3001870084e1784a5ca)
2019-11-22 18:29:49 +01:00
Marios Trivyzas fd1bb4a33a SQL: Fix issue with mins & hours for DATEDIFF ()
Previously, DATEDIFF for minutes and hours was doing a
rounding calculation using all the time fields (secs, msecs/micros/nanos).
Instead it should first truncate the 2 dates to the respective field (mins or hours)
zeroing out all the more detailed time fields and then make the subtraction.

(cherry picked from commit 124cd18e20429e19d52fd8dc383827ea5132d428)
2019-11-19 14:25:28 +01:00
Marios Trivyzas 7c3198ba44
SQL: [Tests] Mute testReplaceChildren for Pivot ()
Temporarily "mute" the testReplaceChildren for Pivot since it leads to
failing tests for some seeds, since the new child doesn't respond to a
valid data type.

Relates to 

(cherry picked from commit 6200a2207b9a4264d2f3fc976577323c7e084317)
2019-11-14 11:30:33 +01:00
Andrei Stefan 2c73c7dfe3 SQL: binary communication implementation for drivers and the CLI ()
* Introduce binary_format request parameter (binary.format for JDBC) to disable binary
communication between clients (jdbc/odbc) and server.
* for CLI - "binary" command line parameter (or -b) is introduced. Default value is "true".
* binary communication (cbor) is enabled by default
* disabling request parameter introduced for debugging purposes only

(cherry picked from commit f96a5ca61cb9fad9ed59357320af20e669348ce7)
2019-10-31 20:39:41 -04:00
emasab 185e067442 SQL: Failing Group By queries due to different ExpressionIds ()
Fix an issue that arises from the use of ExpressionIds as keys in a lookup map
that helps the QueryTranslator to identify the grouping columns. The issue is
that the same expression in different parts of the query (SELECT clause and GROUP BY clause)
ends up with different ExpressionIds so the lookup fails. So, instead of ExpressionIds
use the hashCode() of NamedExpression.

Fixes: 
Fixes: 
Fixes: 
Fixes: 
Fixes: 
Fixes: 
Fixes: 
Fixes: 

Fixes: 
(cherry picked from commit 3c38ea555984fcd2c6bf9e39d0f47a01b09e7c48)
2019-10-31 14:49:16 +01:00
Przemyslaw Gomulka aaa6209be6
[7.x] [Java.time] Calculate week of a year with ISO rules BACKPORT() ()
Reverting the change introducing IsoLocal.ROOT and introducing IsoCalendarDataProvider that defaults start of the week to Monday and requires minimum 4 days in first week of a year. This extension is using java SPI mechanism and defaults for Locale.ROOT only.
It require jvm property java.locale.providers to be set with SPI,COMPAT

closes 
backport 
2019-10-23 17:39:38 +02:00
Andrei Stefan 3233b59b68 Add "format" to "range" queries resulted from optimizing a logical AND ()
(cherry picked from commit 020939a9bd5b34c6d540faa8b3a67b740d661be3)
2019-10-22 10:17:37 +03:00
Marios Trivyzas 3233bce8cb
SQL: Fix issue with negative literels and parentheses ()
Previously when a numeric literal was enclosed in parentheses and then
negated, the negation was lost and the number was considered positive, e.g.:
`-(5)` was considered as `5` instead of `-5`
`- ( (1.28) )` was considered as `1.28` instead of `-1.28`

Fixes: 

(cherry picked from commit 4dee4bf3b34081062ba2e28ab8524a066812a180)
2019-10-16 12:56:35 +02:00
Marios Trivyzas 7fddf198b7 SQL: Implement DATEDIFF function ()
Implement DATEDIFF/TIMESTAMPDIFF function as per the MS-SQL spec:
https://docs.microsoft.com/en-us/sql/t-sql/functions/datediff-transact-sql?view=sql-server-2017
which allows a user to substract two date/datetime fields and return the
difference in the date/time unit specified.

Closes: 
(cherry picked from commit 745699f38dc8222670ffd65b66df33b5da39040b)
2019-10-15 15:12:11 +02:00
Marios Trivyzas 6589617a51
SQL: Fix arg verification for DateAddProcessor ()
Previously, the safety check for the 2nd argument of the DateAddProcessor was
restricting it to Integer which was wrong since we allow all non-rational
numbers, so it's changed to a Number check as it's done in other cases.

Enhanced some tests regarding the check for an integer (non-rational
argument).

(cherry picked from commit 0516b6eaf5eb98fa5bd087c3fece80139a6b118e)
2019-10-15 12:52:11 +02:00
Marios Trivyzas 65717f6f42 SQL: Fix Nullability of DATEADD ()
Previously, Nullability was set to UNKNOWN instead of TRUE which
resulted on QueryFolder not correctly folding to NULL if any of the args
was null.

Remove the overriding nullable() also for DatePart/DateTrunc to allow
delegation the parent class.

(cherry picked from commit 05a7108e133b5ae7bec2257db5ae2d30ad926ee2)
2019-10-12 13:25:08 +02:00
Marios Trivyzas 59b3294bc9 SQL: Implement DATEADD function ()
Implement DATEADD/TIMESTAMPADD function as per the MS-SQL spec:
https://docs.microsoft.com/en-us/sql/t-sql/functions/dateadd-transact-sql?view=sql-server-2017
which allows a user to add/subtract specified number of specified units
to/from a date/datetime field/expression.

Closes: 
(cherry picked from commit e624bc281bebb4bbe0b0c2e0a8cbc712e50097a8)
2019-10-10 16:22:13 +02:00
Marios Trivyzas c1f30e34ff SQL: Refactor binary date time functions ()
Refactor DateTrunc and DatePart to use separate Pipe classes which
allows the removal of the BinaryDateOperation enum.

(cherry picked from commit a6075e7718dff94a90dbc0795dd924dcb7641092)
2019-10-10 13:52:41 +02:00
Andrei Stefan 6a4bf5de2c SQL: make date/datetime and interval types compatible in conditional functions ()
(cherry picked from commit 6ff953e6396d7cc90640419aee5d036954e2eae3)
2019-10-10 13:58:35 +03:00
Andrei Stefan 75a7daae73 SQL: use calendar interval of 1y instead of fixed interval for grouping by YEAR and HISTOGRAMs ()
(cherry picked from commit 55f5463eee4ecea3537df4b34645f1d87472a802)
2019-10-09 11:51:35 +03:00
Marios Trivyzas e698e68f06 SQL: Allow whitespaces in escape patterns ()
Previously, we supported only the format `{fn <FUNCTION_NAME>()}`
but other DBs like MSSQL, DB2, MariaDB/MySQL alos allow whitespaces
between `{` and `fn`. Furhermore, also some applications - like PowerBI -
generate escape sequences with spaces: `select { fn name(params) } etc.`

Add support for white spaces between `{` and the escape pattern definition
like `fn`, `ts`, `d`, `guid` etc.

Closes: 

(cherry picked from commit 08a22d0b393f4a76c52dabc5e7b9cafcc19c30ca)
2019-10-07 15:05:02 +02:00
Andrei Stefan a46f312ded SQL: fix multi full-text functions usage with aggregate functions ()
* Skip functions involving full-text predicates when replacing multiple
aggregate functions with "stats" or "matrix_stats" aggregations.

(cherry picked from commit bb14ba83128dfb7a70f825ea08b1524072fb9ad0)
2019-10-04 16:27:22 +03:00
Alpar Torok 0a14bb174f Remove eclipse conditionals ()
* Remove eclipse conditionals

We used to have some meta projects with a `-test` prefix because
historically eclipse could not distinguish between test and main
source-sets and could only use a single classpath.
This is no longer the case for the past few Eclipse versions.

This PR adds the necessary configuration to correctly categorize source
folders and libraries.
With this change eclipse can import projects, and the visibility rules
are correct e.x. auto compete doesn't offer classes from test code or
`testCompile` dependencies when editing classes in `main`.

Unfortunately the cyclic dependency detection in Eclipse doesn't seem to
take the difference between test and non test source sets into account,
but since we are checking this in Gradle anyhow, it's safe to set to
`warning` in the settings. Unfortunately there is no setting to ignore
it.

This might cause problems when building since Eclipse will probably not
know the right order to build things in so more wirk might be necesarry.
2019-10-03 11:55:00 +03:00
Henning Andersen 42453aec96 Fix XPackPlugin usages in tests ()
XPackPlugin holds data in statics and can only be initialized once. This
caused tests to fail primarily when running with a low max-workers.

Replaced usages with the LocalStateCompositeXPackPlugin, which handles
this properly for testing.
2019-10-02 12:36:02 +02:00
Marios Trivyzas f792dbf239 SQL: Implement DATE_PART function ()
DATE_PART(<datetime unit>, <date/datetime>) is a function that allows
the user to extract the specified unit from a date/datetime field
similar to the EXTRACT (<datetime unit> FROM <date/datetime>) but
with different names and aliases for the units and it also provides more
options like `DATE_PART('tzoffset', datetimeField)`.

Implemented following the SQL server's spec: https://docs.microsoft.com/en-us/sql/t-sql/functions/datepart-transact-sql?view=sql-server-2017
with the difference that the <datetime unit> argument is either a
literal single quoted string or gets a value from a table field, whereas
in SQL server keywords are used (unquoted identifiers) and it's not
possible to use a value coming for a table column.

Closes: 
(cherry picked from commit ead743d3579eb753fd314d4a58fae205e465d72e)
2019-10-01 16:28:27 +03:00
emasab 87156ad93b
SQL: Fix issue with duplicate columns in SELECT ()
Previously, if a column (field, scalar, alias) appeared more than once in the
SELECT list, the value was returned only once (1st appearance) in each row.

Fixes: 

(cherry picked from commit 097ea36581a751605fc4f2088319d954ce35b5d1)
2019-09-30 15:56:29 +03:00
Marios Trivyzas 01623f9f1c
SQL: Add alias DATETRUNC to DATE_TRUNC function ()
To be on the safe side in terms of use cases also add the alias
DATETRUNC to the DATE_TRUNC function.

Follows: 

(cherry picked from commit 9ac223cb1fc66486f86e218fa785a32b61e9bacc)
2019-09-27 15:38:51 +03:00
Costin Leau b29a2cb360 SQL: Check case where the pivot limit is reached ()
In some cases, the fetch size affects the way the groups are returned
causing the last page to go beyond the limit. Add dedicated check to
prevent extra data from being returned.

Fix 

(cherry picked from commit f4c29646f097bbd29855300342823ef4cef61c05)
2019-09-26 22:32:42 +03:00
Igor Motov ae202fda21 SQL: Add support for shape type ()
Enables support for Cartesian geometries shape type. We still need to
decide how to handle the distance function since it is currently using
the haversine distance formula and returns results in meters, which
doesn't make any sense for Cartesian geometries.

Closes 
Relates to 
2019-09-26 09:47:42 -04:00
Costin Leau a610503783 SQL: Add PIVOT support ()
Add initial PIVOT support for transforming a regular table into a
statistics table around an arbitrary pivoting column:

SELECT * FROM
 (SELECT languages, country, salary, FROM mp)
 PIVOT (AVG(salary) FOR countries IN ('NL', 'DE', 'ES', 'RO', 'US'))

In the current implementation PIVOT allows only one aggregation however
this restriction is likely to be lifted in the future.
Also not all aggregations are working, in particular MatrixStats are not yet supported.

(cherry picked from commit d91263746a222915c570d4a662ec48c1d6b4f583)
2019-09-23 21:04:13 +03:00
Costin Leau 92e518e789 SQL: Properly handle indices with no/empty mapping ()
When encountering only indices with empty mapping, the IndexResolver
throws an exception as it expects to find at least one entry.
This commit fixes this case so that an empty mapping is returned.

Fix 

(cherry picked from commit 5f4f5807acb93b5fab36718c092c328977a396b6)
2019-09-17 16:01:22 +03:00
Costin Leau 683b5fdeca SQL: Support queries with HAVING over SELECT ()
Handle queries with implicit GROUP BY where the aggregation is not in
the projection/SELECT but inside the filter/HAVING such as:

SELECT 1 FROM x HAVING COUNT(*) > 0

The engine now properly identifies the case and handles it accordingly.

Fix 

(cherry picked from commit fa53ca05d8219c27079b50b4a5b7aeb220c7cde2)
2019-09-17 11:14:39 +03:00
Andrei Stefan 40e9353947 SQL: use the correct data type for types conversion ()
(cherry picked from commit 3e25db2f302c3aafe27e4d8d4fb1743401d85e6d)
2019-09-16 15:36:17 +03:00
Marios Trivyzas d956509394 SQL: Implement DATE_TRUNC function ()
DATE_TRUNC(<truncate field>, <date/datetime>) is a function that allows
the user to truncate a timestamp to the specified field by zeroing out
the rest of the fields. The function is implemented according to the
spec from PostgreSQL: https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC

Closes: 
(cherry picked from commit b37e96712db1aace09f17b574eb02ff6b942a297)
2019-09-11 21:41:02 +03:00
Marios Trivyzas 0963e78164 SQL: Fix issue with common type resolution ()
Many scalar functions try to find out the common type between their
arguments in order to set it as their return time, e.g.:
for `float + double` the common type which is set as the return type
of the + operation is `double`.

Previously, for data types TEXT and KEYWORD (string data types) there
was no common data type found and null was returned causing NPEs when
the function was trying to resolve the return data type.

Fixes: 
(cherry picked from commit 291017d69dfc810707c3c7c692f5a50af431b790)
2019-09-11 19:10:15 +03:00
Andrei Stefan 7b26a8c041 Use `null` schema response for `SYS TABLES` command. ()
(cherry picked from commit a6152f42a47a1ccd668e5892778c8bd2d3a78c4c)
2019-09-07 09:24:54 +03:00
Andrei Stefan 7cf100ba07 SQL: fix scripting for grouped by datetime functions ()
* Fix issue with painless scripting not being correctly generated when
datetime functions are used for GROUPing of an INTERVAL operation.

(cherry picked from commit cb92828e8ec9d9d241bd6189e5835fd99f8b9a44)
2019-09-07 09:24:53 +03:00
Marios Trivyzas fd0affb503
SQL: Fix issue with IIF function when condition folds ()
Previously, when the condition (1st argument) of the IIF function could
be evaluated (folded) to false, the `IfConditional` was eliminated which
caused `IndexOutOfBoundsException` to be thrown when `info()` and
`resolveType()` methods where called.

Fixes: 

(cherry picked from commit 9a885a3ac47bc8f52c07770d1d8d670ce0af1e59)
2019-09-04 10:32:49 +03:00
Marios Trivyzas 3bee647e5b
SQL: Fix issue with DataType for CASE with NULL ()
Previously, if the DataType of all the WHEN conditions of a CASE
statement is NULL, then it was set to NULL even if the ELSE clause
has a non-NULL data type, e.g.:
```
CASE WHEN a = 1 THEN NULL
           WHEN a = 5 THEN NULL
ELSE 'foo'
```

Fixes: 
(cherry picked from commit 8c1012efbbd3a300afd0dfb9b18250f15ea753f9)
2019-09-02 11:17:24 +03:00
Costin Leau 0f51dd69cb SQL: Improve serialization of SQL processors ()
Encapsulate the serialization/deserialization of SQL client classes.
Make configuration specific parameters (such as ZoneId) generic just
like the version and remove the need for consumer classes to manage them
individually.
This is not only consistent but also provides significant savings in the
cursor.

Fix 

(cherry picked from commit 5c844798045d7baa0d932289d2e3d1607ba6a9a4)
2019-08-20 11:50:47 +03:00
Costin Leau 1cd58c8ea8 SQL: Break TextFormatter/Cursor dependency ()
Improve the initialization and state passing of TextFormatter in CLI
and TEXT mode by leveraging the Page listener hook. Additionally
simplify the code inside RestSqlQueryAction.

(cherry picked from commit a56db2fa119cf9e8748723e19f1fc9f6a8afe5fc)
2019-08-17 00:16:08 +03:00
Costin Leau 96883dd028 SQL: Refactor away the cycle between Rowset and Cursor ()
Improve encapsulation of pagination of rowsets by breaking the cycle
between cursor and associated rowset implementation, all logic now
residing inside each cursor implementation.

(cherry picked from commit be8fe0a0ce562fe732fae12a0b236b5731e4638c)
2019-08-17 00:16:05 +03:00
Igor Motov 98c850c08b
Geo: Change order of parameter in Geometries to lon, lat 7.x ()
Changes the order of parameters in Geometries from lat, lon to lon, lat
and moves all Geometry classes are moved to the
org.elasticsearch.geomtery package.

Backport of 

Closes 
2019-08-16 14:42:02 -04:00
Andrei Stefan 30a0711777 Remove deprecated use of "interval" method, in favor of "fixedInterval". ()
(cherry picked from commit 3fef65160f9e61883e9f8f7f345b814f945e2f4b)
2019-08-16 15:03:43 +03:00
Andrei Stefan adf8e20021
SQL: adds format parameter to range queries for constant date comparisons ()
* Add format parameter to the range queries built for CURRENT_* functions
used in comparison conditions
* Use range queries for date fields equality/non-equality as well.

(cherry picked from commit c1e81e90f937ee5a002524d632bfce74d76962f9)
2019-08-13 23:04:30 +03:00
Armin Braun a9e1402189
Remove Settings from BaseRestRequest Constructor () ()
* Resolving the todo, cleaning up the unused `settings` parameter
* Cleaning up some other minor dead code in affected classes
2019-08-12 05:14:45 +02:00
Andrei Stefan 740d58fd46
SQL: Uniquely named inner_hits sections for each nested field condition ()
* Name each inner_hits section of nested queries differently and extract and combine the multiple values it generates into a single list.
This also introduces a limitation (its origin it's with Elasticsearch
though) on the sorting capabilities when the sorting is based on the
nested fields filtered: only one of the conditions applied to nested
documents will be used in the nested sorting.

(cherry picked from commit cfc5cf68f6e83b07bb9006986d0903d6be418ec6)
2019-08-09 00:22:49 +03:00
Andrei Stefan 2633d11eb7
Switch from using docvalue_fields to extracting values from _source () ()
* Switch from using docvalue_fields to extracting values from _source
where applicable. Doing this means parsing the _source and handling the
numbers parsing just like Elasticsearch is doing it when it's indexing
a document.
* This also introduces a minor limitation: aliases type of fields that
are NOT part of a tree of sub-fields will not be able to be retrieved
anymore. field_caps API doesn't shed any light into a field being an
alias or not and at _source parsing time there is no way to know if a
root field is an alias or not. Fields of the type "a.b.c.alias" can be
extracted from docvalue_fields, only if the field they point to can be
extracted from docvalue_fields. Also, not all fields in a hierarchy of
fields can be evaluated to being an alias.

(cherry picked from commit 8bf8a055e38f00df5f49c8d97f632f69d6e00c2c)
2019-07-25 10:02:41 +03:00
Andrei Stefan 04cb3aebd5 Use hasValue() methods from aggregations' InspectionHelpers ()
Use InspectionHelper classes to decide if the aggregations should return null (in case there is no value) or the value itself.

(cherry picked from commit dafd7b039b0da072750e8f57e7572d24f7aad44a)
2019-07-24 16:14:45 +03:00
Tal Levy 7c84636029
Remove StreamOutput #writeOptionalStreamable and #writeStreamableList () ()
remove usages of writeOptionalStreamable and writeStreambaleList

relates .
2019-07-19 15:55:53 -07:00
Ryan Ernst 60785a9fa8
Convert several direct uses of Streamable to Writeable () ()
This commit converts several utility classes that implement Streamable
to have StreamInput constructors. It also adds a default version of
readFrom to Streamable so that overriding to throw UOE is not necessary.

relates 
2019-07-18 21:25:44 -07:00
Ryan Ernst edd26339c5
Convert remaining request classes in xpack core to writeable.reader () ()
This commit converts all remaining classes extending ActionRequest
in xpack core to have a StreamInput constructor.

relates 
2019-07-18 01:11:45 -07:00