234 lines
11 KiB
Plaintext
234 lines
11 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[sql-limitations]]
|
|
== SQL Limitations
|
|
|
|
[float]
|
|
[[large-parsing-trees]]
|
|
=== Large queries may throw `ParsingExpection`
|
|
|
|
Extremely large queries can consume too much memory during the parsing phase, in which case the {es-sql} engine will
|
|
abort parsing and throw an error. In such cases, consider reducing the query to a smaller size by potentially
|
|
simplifying it or splitting it into smaller queries.
|
|
|
|
[float]
|
|
[[sys-columns-describe-table-nested-fields]]
|
|
=== Nested fields in `SYS COLUMNS` and `DESCRIBE TABLE`
|
|
|
|
{es} has a special type of relationship fields called `nested` fields. In {es-sql} they can be used by referencing their inner
|
|
sub-fields. Even though `SYS COLUMNS` in non-driver mode (in the CLI and in REST calls) and `DESCRIBE TABLE` will still display
|
|
them as having the type `NESTED`, they cannot be used in a query. One can only reference its sub-fields in the form:
|
|
|
|
[source, sql]
|
|
--------------------------------------------------
|
|
[nested_field_name].[sub_field_name]
|
|
--------------------------------------------------
|
|
|
|
For example:
|
|
|
|
[source, sql]
|
|
--------------------------------------------------
|
|
SELECT dep.dep_name.keyword FROM test_emp GROUP BY languages;
|
|
--------------------------------------------------
|
|
|
|
[float]
|
|
=== Scalar functions on nested fields are not allowed in `WHERE` and `ORDER BY` clauses
|
|
{es-sql} doesn't support the usage of scalar functions on top of nested fields in `WHERE`
|
|
and `ORDER BY` clauses with the exception of comparison and logical operators.
|
|
|
|
For example:
|
|
|
|
[source, sql]
|
|
--------------------------------------------------
|
|
SELECT * FROM test_emp WHERE LENGTH(dep.dep_name.keyword) > 5;
|
|
--------------------------------------------------
|
|
|
|
and
|
|
|
|
[source, sql]
|
|
--------------------------------------------------
|
|
SELECT * FROM test_emp ORDER BY YEAR(dep.start_date);
|
|
--------------------------------------------------
|
|
|
|
are not supported but:
|
|
|
|
[source, sql]
|
|
--------------------------------------------------
|
|
SELECT * FROM test_emp WHERE dep.start_date >= CAST('2020-01-01' AS DATE) OR dep.dep_end_date IS NULL;
|
|
--------------------------------------------------
|
|
|
|
is supported.
|
|
|
|
[float]
|
|
=== Multi-nested fields
|
|
|
|
{es-sql} doesn't support multi-nested documents, so a query cannot reference more than one nested field in an index.
|
|
This applies to multi-level nested fields, but also multiple nested fields defined on the same level. For example, for this index:
|
|
|
|
[source, sql]
|
|
----------------------------------------------------
|
|
column | type | mapping
|
|
----------------------+---------------+-------------
|
|
nested_A |STRUCT |NESTED
|
|
nested_A.nested_X |STRUCT |NESTED
|
|
nested_A.nested_X.text|VARCHAR |KEYWORD
|
|
nested_A.text |VARCHAR |KEYWORD
|
|
nested_B |STRUCT |NESTED
|
|
nested_B.text |VARCHAR |KEYWORD
|
|
----------------------------------------------------
|
|
|
|
`nested_A` and `nested_B` cannot be used at the same time, nor `nested_A`/`nested_B` and `nested_A.nested_X` combination.
|
|
For such situations, {es-sql} will display an error message.
|
|
|
|
[float]
|
|
=== Paginating nested inner hits
|
|
|
|
When SELECTing a nested field, pagination will not work as expected, {es-sql} will return __at least__ the page size records.
|
|
This is because of the way nested queries work in {es}: the root nested field will be returned and it's matching inner nested fields as well,
|
|
pagination taking place on the **root nested document and not on its inner hits**.
|
|
|
|
[float]
|
|
[[normalized-keyword-fields]]
|
|
=== Normalized `keyword` fields
|
|
|
|
`keyword` fields in {es} can be normalized by defining a `normalizer`. Such fields are not supported in {es-sql}.
|
|
|
|
[float]
|
|
=== Array type of fields
|
|
|
|
Array fields are not supported due to the "invisible" way in which {es} handles an array of values: the mapping doesn't indicate whether
|
|
a field is an array (has multiple values) or not, so without reading all the data, {es-sql} cannot know whether a field is a single or multi value.
|
|
When multiple values are returned for a field, by default, {es-sql} will throw an exception. However, it is possible to change this behavior through `field_multi_value_leniency` parameter in REST (disabled by default) or
|
|
`field.multi.value.leniency` in drivers (enabled by default).
|
|
|
|
[float]
|
|
=== Sorting by aggregation
|
|
|
|
When doing aggregations (`GROUP BY`) {es-sql} relies on {es}'s `composite` aggregation for its support for paginating results.
|
|
However this type of aggregation does come with a limitation: sorting can only be applied on the key used for the aggregation's buckets.
|
|
{es-sql} overcomes this limitation by doing client-side sorting however as a safety measure, allows only up to *512* rows.
|
|
|
|
It is recommended to use `LIMIT` for queries that use sorting by aggregation, essentially indicating the top N results that are desired:
|
|
|
|
[source, sql]
|
|
--------------------------------------------------
|
|
SELECT * FROM test GROUP BY age ORDER BY COUNT(*) LIMIT 100;
|
|
--------------------------------------------------
|
|
|
|
It is possible to run the same queries without a `LIMIT` however in that case if the maximum size (*10000*) is passed,
|
|
an exception will be returned as {es-sql} is unable to track (and sort) all the results returned.
|
|
|
|
[float]
|
|
=== Using aggregation functions on top of scalar functions
|
|
|
|
Aggregation functions like <<sql-functions-aggs-min,`MIN`>>, <<sql-functions-aggs-max,`MAX`>>, etc. can only be used
|
|
directly on fields, and so queries like `SELECT MAX(abs(age)) FROM test` are not possible.
|
|
|
|
[float]
|
|
=== Using a sub-select
|
|
|
|
Using sub-selects (`SELECT X FROM (SELECT Y)`) is **supported to a small degree**: any sub-select that can be "flattened" into a single
|
|
`SELECT` is possible with {es-sql}. For example:
|
|
|
|
["source","sql",subs="attributes,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{sql-specs}/docs/docs.csv-spec[limitationSubSelect]
|
|
--------------------------------------------------
|
|
|
|
The query above is possible because it is equivalent with:
|
|
|
|
["source","sql",subs="attributes,macros"]
|
|
--------------------------------------------------
|
|
include-tagged::{sql-specs}/docs/docs.csv-spec[limitationSubSelectRewritten]
|
|
--------------------------------------------------
|
|
|
|
But, if the sub-select would include a `GROUP BY` or `HAVING` or the enclosing `SELECT` would be more complex than `SELECT X
|
|
FROM (SELECT ...) WHERE [simple_condition]`, this is currently **un-supported**.
|
|
|
|
[float]
|
|
[[first-last-agg-functions-having-clause]]
|
|
=== Using <<sql-functions-aggs-first, `FIRST`>>/<<sql-functions-aggs-last,`LAST`>> aggregation functions in `HAVING` clause
|
|
|
|
Using `FIRST` and `LAST` in the `HAVING` clause is not supported. The same applies to
|
|
<<sql-functions-aggs-min,`MIN`>> and <<sql-functions-aggs-max,`MAX`>> when their target column
|
|
is of type <<keyword, `keyword`>> as they are internally translated to `FIRST` and `LAST`.
|
|
|
|
[float]
|
|
[[group-by-time]]
|
|
=== Using TIME data type in GROUP BY or <<sql-functions-grouping-histogram>>
|
|
|
|
Using `TIME` data type as a grouping key is currently not supported. For example:
|
|
|
|
[source, sql]
|
|
-------------------------------------------------------------
|
|
SELECT count(*) FROM test GROUP BY CAST(date_created AS TIME);
|
|
-------------------------------------------------------------
|
|
|
|
On the other hand, it can still be used if it's wrapped with a scalar function that returns another data type,
|
|
for example:
|
|
|
|
[source, sql]
|
|
-------------------------------------------------------------
|
|
SELECT count(*) FROM test GROUP BY MINUTE((CAST(date_created AS TIME));
|
|
-------------------------------------------------------------
|
|
|
|
`TIME` data type is also currently not supported in histogram grouping function. For example:
|
|
|
|
[source, sql]
|
|
-------------------------------------------------------------
|
|
SELECT HISTOGRAM(CAST(birth_date AS TIME), INTERVAL '10' MINUTES) as h, COUNT(*) FROM t GROUP BY h
|
|
-------------------------------------------------------------
|
|
|
|
[float]
|
|
[[geo-sql-limitations]]
|
|
=== Geo-related functions
|
|
|
|
Since `geo_shape` fields don't have doc values these fields cannot be used for filtering, grouping or sorting.
|
|
|
|
By default,`geo_points` fields are indexed and have doc values. However only latitude and longitude are stored and
|
|
indexed with some loss of precision from the original values (4.190951585769653E-8 for the latitude and
|
|
8.381903171539307E-8 for longitude). The altitude component is accepted but not stored in doc values nor indexed.
|
|
Therefore calling `ST_Z` function in the filtering, grouping or sorting will return `null`.
|
|
|
|
[float]
|
|
[[fields-from-source]]
|
|
=== Retrieving from `_source`
|
|
|
|
Most of {es-sql}'s columns are retrieved from the document's `_source` and there is no attempt to get the columns content from
|
|
`docvalue_fields` not even in the case <<mapping-source-field,`_source`>> field is disabled in the mapping explicitly.
|
|
If a column, for which there is no source stored, is asked for in a query, {es-sql} will not return it. Field types that don't follow
|
|
this restriction are: `keyword`, `date`, `scaled_float`, `geo_point`, `geo_shape` since they are NOT returned from `_source` but
|
|
from `docvalue_fields`.
|
|
|
|
[float]
|
|
[[fields-from-docvalues]]
|
|
=== Retrieving from `docvalue_fields`
|
|
|
|
When the number of columns retrievable from `docvalue_fields` is greater than the configured <<dynamic-index-settings,`index.max_docvalue_fields_search` setting>>
|
|
the query will fail with `IllegalArgumentException: Trying to retrieve too many docvalue_fields` error. Either the mentioned {es}
|
|
setting needs to be adjusted or fewer columns retrievable from `docvalue_fields` need to be selected.
|
|
|
|
[float]
|
|
[[aggs-in-pivot]]
|
|
=== Aggregations in the <<sql-syntax-pivot, `PIVOT`>> clause
|
|
|
|
The aggregation expression in <<sql-syntax-pivot, `PIVOT`>> will currently accept only one aggregation. It is thus not possible to obtain multiple aggregations for any one pivoted column.
|
|
|
|
[float]
|
|
[[subquery-in-pivot]]
|
|
=== Using a subquery in <<sql-syntax-pivot, `PIVOT`>>'s `IN`-subclause
|
|
|
|
The values that the <<sql-syntax-pivot, `PIVOT`>> query could pivot must be provided in the query as a list of literals; providing a subquery instead to build this list is not currently supported. For example, in this query:
|
|
|
|
[source, sql]
|
|
-------------------------------------------------------------
|
|
SELECT * FROM test_emp PIVOT (SUM(salary) FOR languages IN (1, 2))
|
|
-------------------------------------------------------------
|
|
|
|
the `languages` of interest must be listed explicitly: `IN (1, 2)`. On the other hand, this example would **not work**:
|
|
|
|
[source, sql]
|
|
-------------------------------------------------------------
|
|
SELECT * FROM test_emp PIVOT (SUM(salary) FOR languages IN (SELECT languages FROM test_emp WHERE languages <=2 GROUP BY languages))
|
|
-------------------------------------------------------------
|