Commit Graph

633 Commits

Author SHA1 Message Date
Hicham Mallah 5b32d112e1
SQL: Fix issues with GROUP BY queries (#41964)
Translate to an agg query even if only literals are selected,
so that the correct number of rows is returned (number of buckets).

Fix issue with key only in GROUP BY (not in select) and WHERE clause:
Resolve aggregates and groupings based on the child plan which holds
the info info for all the fields of the underlying table.

Fixes: #41951
Fixes: #41413
(cherry picked from commit 45b85809678b34a448639a420b97e25436ae851f)
2020-02-15 10:38:24 +01:00
Marios Trivyzas 51e74be1bb
SQL: [Tests] Add tests for fixed issues (#52335)
Add tests to verify behaviour for
fixed issues: #33724 & #38306

(cherry picked from commit 89fb6753a9db9484a5622417cd4ffea9af0347ad)
2020-02-14 11:23:30 +01:00
Julie Tibshirani 0d7165a40b Standardize naming of fetch subphases. (#52171)
This commit makes the names of fetch subphases more consistent:
* Now the names end in just 'Phase', whereas before some ended in
  'FetchSubPhase'. This matches the query subphases like AggregationPhase.
* Some names include 'fetch' like FetchScorePhase to avoid ambiguity about what
  they do.
2020-02-13 13:00:46 -08:00
Costin Leau 5373a77fb9 QL: Extract common Failure class (#52281)
Shared across SQL and EQL

(cherry picked from commit 1aeda20d3ec3d6c885de03c6043dd1e8eab9f230)
2020-02-13 14:35:15 +02:00
Jay Modi 5bcc6fce5c
Remove DeprecationLogger from route objects (#52285)
This commit removes the need for DeprecatedRoute and ReplacedRoute to
have an instance of a DeprecationLogger. Instead the RestController now
has a DeprecationLogger that will be used for all deprecated and
replaced route messages.

Relates #51950
Backport of #52278
2020-02-12 15:05:41 -07:00
Bogdan Pintea 5dfe27601e
SQL: supplement input checks on received request parameters (#52229) (#52277)
* Add more checks around parameter conversions

This commit adds two necessary verifications on received parameters:
- it checks the validity of the parameter's data type: if the declared
data type is resolved to an ES or Java type;
- it checks if the returned converter is non-null (i.e. a conversion is
possible) and generates an appropriate exception otherwise.

(cherry picked from commit eda30ac9c69383165324328c599ace39ac064342)
2020-02-12 19:45:12 +01:00
Andrei Stefan a3ebacfcf3
52169 & 52172 7x backport (#52256)
* Extract common optimizer tests (#52169)

(cherry picked from commit e5ad72bc22e9ec0686ab582195f0032efcb880bf)

* Hook in the optimizer rules (#52172)

(cherry picked from commit 1f90d8cc56052fbf2af604e72f9f5ca73f5e75d5)
2020-02-12 11:20:03 +02:00
Marios Trivyzas daab242c75
SQL: Fix ORDER BY on aggregates and GROUPed BY fields (#51894)
Previously, in the in-memory sorting module
`LocalAggregationSorterListener` only the aggregate functions where used
(grabbed by the `sortingColumns`). As a consequence, if the ORDER BY
was also using columns of the GROUP BY clause, (especially in the case
of higher priority - before the aggregate functions) wrong results were
produced. E.g.:
```
SELECT gender, MAX(salary) AS max FROM test_emp
GROUP BY gender
ORDER BY gender, max
```

Add all columns of the ORDER BY to the `sortingColumns` so that the
`LocalAggregationSorterListener` can use the correct comparators in
the underlying PriorityQueue used to implement the in-memory sorting.

Fixes: #50355
(cherry picked from commit be680af11c823292c2d115bff01658f7b75abd76)
2020-02-12 09:38:47 +01:00
Marios Trivyzas 204d086266 SQL: Fix issue with timezone when paginating (#52101)
Previously, when the specified (or default) fetchSize led to
subsequent HTTP requests and the usage of cursors, those subsequent
were no longer using the client timezone specified in the initial
SQL query. As a consequence, Even though the query is executed once
(with the correct timezone) the processing of the query results by
the HitExtractors in the next pages was done using the default
timezone Z. This could lead to incorrect results.

Fix the issue by correctly using the initially specified timezone,
which is found in the deserialisation of the cursor string.

Fixes: #51258
(cherry picked from commit 8f7afbdeb9295999b48a6c36db5b31cbe0cee432)
2020-02-11 15:27:56 +01:00
Marios Trivyzas 6b600855a9
SQL: Make parsing of date more lenient (#52137)
Make the parsing of date more lenient

- as an escaped literal: `{d '2020-02-10[[T| ]10:20[:30][.123456789][tz]]'}`
- cast a string to a date: `CAST(2020-02-10[[T| ]10:20[:30][.123456789][tz]]' AS DATE)`

Closes: #49379
(cherry picked from commit 5863b27500d5e7f6cdd8c6c62b09b84e53ca724a)
2020-02-10 21:47:00 +01:00
Bogdan Pintea 7b58ed0dd7
Fix milliseconds handling in intervals (#51675) (#52156)
This fixes:

- the parsing of milliseconds in intervals: everything past the . used to be converted as-is to milliseconds, with no normalisation of the unit; thus, a value of .23 ended up as 23 millis in the interval, instead of 230.
- the printing of a trailing .0, in case the interval lacks the fractional part;
- tests generating a random millisecond value used to simply print it in the string about to be evaluated without a necessary front-filling of 0[s], where the amount was below 100/10.

(The combination of first and last issues above, plus statistical "luck" made the incorrect handling pass the tests.)

(cherry picked from commit 4de8c64f63ee37c1bcfdb9b9d3a07d09be243222)
2020-02-10 19:24:26 +01:00
Marios Trivyzas 27265f032a SQL: Enhance timestamp escaped literal parsing (#52097)
Allow also whitespace ` ` (together with `T`) as a separator between
date and time parts of the timestamp string. E.g.:
```
{ts '2020-02-08 12.10.45'}
```
or
```
{ts '2020-02-08T12.10.45'}
```

Fixes: #46069
(cherry picked from commit 07c977023fb8ceab5991c359a6cbfe07beaad9bb)
2020-02-10 11:24:55 +01:00
Andrei Stefan fa4dcd50d9 Extract common optimization rules for QL (#52054) (#52132)
(cherry picked from commit ee43115531234c2d955193ce0c9c268e1f02ab43)
2020-02-10 11:48:45 +02:00
Ignacio Vera 80e3c97210 Upgrade to lucene-8.5.0-snapshot-d62f6307658 (#52039) (#52130) 2020-02-10 10:13:22 +01:00
Jay Modi 3edadfefd0 RestHandlers declare handled routes (#52123)
This commit changes how RestHandlers are registered with the
RestController so that a RestHandler no longer needs to register itself
with the RestController. Instead the RestHandler interface has new
methods which when called provide information about the routes
(method and path combinations) that are handled by the handler
including any deprecated and/or replaced combinations.

This change also makes the publication of RestHandlers safe since they
no longer publish a reference to themselves within their constructors.

Closes #51622

Co-authored-by: Jason Tedor <jason@tedor.me>

Backport of #51950
2020-02-09 22:48:32 -07:00
Marios Trivyzas 3e7f939f63
SQL: [Tests] Add more tests for aggs and literals (#52086)
Add some more tests where more than one literal is selected,
unaliased and aliased.

Follows: #42121
(cherry picked from commit 405271d408a233e697eb2e9ded3005a71f4df5e7)
2020-02-09 18:01:05 +01:00
Costin Leau 214beed90f QL: move query AST from SQL to QL (#52069)
(cherry picked from commit 59368968b698652352be1bb2a60d5a357a01b978)
2020-02-08 23:10:51 +02:00
Emanuele Sabellico 282e919607 SQL: [Tests] Add integ tests for selecting a literal and an aggregate (#42121)
The related issue regarding aggregation queries where some literals
are also selected together with aggregate function has been fixed
with #49570. Add integration tests to verify the behavior.

Relates to: #41411

(cherry picked from commit 9f414a8d05c75e1a9f8250084f6dcd634d5d78d8)
2020-02-07 19:00:15 +01:00
Andrei Stefan 488944f4a1
SQL: Handle uberjar scenario where the ES jdbc driver file is bundled in another jar (#51856) (#52024)
(cherry picked from commit 6247b0793c9db19a8a9fa6f0164cc14d0debed6e)
2020-02-07 04:15:59 +02:00
Marios Trivyzas 64f9a2089b SQL: [Tests] add tests for literals and GROUP BY (#51878)
Add unit and integration tests where literals are SELECTed
in combination with GROUP BY and possibly aggregate functions.

Relates to #41411 and #34583
which have been fixed.

(cherry picked from commit b97f1ca12675d6ea4772c60578922fe1cc2409ee)
2020-02-05 12:55:56 +01:00
Julie Tibshirani 38ce428831
Create a class to hold field capabilities for one index. (#51844)
Currently, the same class `FieldCapabilities` is used both to represent the
capabilities for one index, and also the merged capabilities across indices. To
help clarify the logic, this PR proposes to create a separate class
`IndexFieldCapabilities` for the capabilities in one index. The refactor will
also help when adding `source_path` information in #49264, since the merged
source path field will have a different structure from the field for a single index.

Individual changes:
* Add a new class IndexFieldCapabilities.
* Remove extra constructor from FieldCapabilities.
* Combine the add and merge methods in FieldCapabilities.Builder.
2020-02-04 11:24:57 -08:00
Ryan Ernst 21224caeaf Remove comparison to true for booleans (#51723)
While we use `== false` as a more visible form of boolean negation
(instead of `!`), the true case is implied and the true value does not
need to explicitly checked. This commit converts cases that have slipped
into the code checking for `== true`.
2020-01-31 16:35:43 -08:00
Mayya Sharipova 42b885f050
Upgrade to lucene-8.5.0-snapshot-3333ce7da6d (#51749)
Backport for #51327
2020-01-31 11:20:15 -05:00
Bogdan Pintea f1173aaa48
SQL: Add optimisations for not-equalities (#51088) (#51700)
* Optimize not-equalities in con-/disjunctions

This commit adds optimisations of not-equalities in conjunctions and
disjunctions:
* for conjunctions, the not-equality can be optimized away when applied
together with a range or inequality, in case the not-equality point
falls outside the domain of the later condition; if its on the boarder,
it will modify the bound, to simply exclude the equality, if present;
otherwise no optimisation can be applied;
* for disjunctions, the not-equals could filter away the ranges and
inequalities, unless these include an equality on the bound, in which
case the entire condition becomes always true, but this would influence
the score() function, so it's been omitted;

* fix aggregations of inequalities in ranges

This commit fixes the loop that aggregates inequalities into ranges:
- it won't advance the outer loop index in case of a merge, since the
current element is removed;
- it will break the inner loop, since comparision against the element
selected in the outer loop can't continue, as it had been removed.



(cherry picked from commit 789724ac2cc726de603849b4eeb8194da7528bcc)
2020-01-30 23:29:39 +01:00
Marios Trivyzas f373020349 SQL: Fix ORDER BY YEAR() function (#51562)
Previously, if YEAR() was used as and ORDER BY argument without being
wrapped with another scalar (e.g. YEAR(birth_date) + 10), no script
ordering was used but instead the underlying field (e.g. birth_date)
was used instead as a performance optimisation. This works correctly if
YEAR() is the only ORDER BY arg but if further args are used as tie
breakers for the ordering wrong results are produced. This is because
2 rows with the different birth_date but on the same year are not tied
as the underlying ordering is on birth_date and not on the
YEAR(birth_date), and the following ORDER BY args are ignored.

Remove this optimisation for YEAR() to avoid incorrect results in
such cases.

As a consequence another bug is revealed: scalar functions on top
of nested fields produce scripted sorting/filtering which is not yet
supported. In such cases no error was thrown but instead all values for
such nested fields were null and were passed to the script implementing
the sorting/filtering, producing incorrect results.

Detect such cases and throw a validation exception.

Fixes: #51224
(cherry picked from commit f41efd6753dc3650a7eabb3e07b02b3b32c5704c)
2020-01-30 15:29:36 +01:00
Marios Trivyzas 285a167c34 SQL: Verify Full-Text Search functions not allowed in SELECT (#51568)
Add a verification that full-text search functions `MATCH()` and `QUERY()`
are not allowed in the SELECT clause, so that a nice error message is
returned to the user early instead of an "ugly" exception.

Fixes: #47446
2020-01-30 13:14:38 +01:00
Jason Tedor 3a7192966a
Check if interface is up for loopback devices only (#51583)
In the SQL with SSL tests, we need to find the interfaces that are up,
are loopback devices, or have a loopback address. If we check if the
device is up first, we can run into situations where the device is a
virtual ethernet device that might have disappeared between us seeing
the device, and checking if it is up. By first checking if the device is
a loopback device or it has a loopback address, then we can avoid
checking if the device is up except for loopback devices and therefore
we can avoid the disappearing virtual ethernet device problem.
2020-01-28 18:38:46 -05:00
Costin Leau e22f501018
QL: Backport project to 7.x (#51497)
* Introduce reusable QL plugin for SQL and EQL (#50815)

Extract reusable functionality from SQL into its own dedicated project QL.
Implemented as a plugin, it provides common components across SQL and the upcoming EQL.

While this commit is fairly large, for the most part it's just a big file move from sql package to the newly introduced ql.

(cherry picked from commit ec1ac0d463bfa12a02c8174afbcdd6984345e8b4)

* SQL: Fix incomplete registration of geo NamedWritables

(cherry picked from commit e295763686f9592976e551e504fdad1d2a3a566d)

* QL: Extend NodeSubclass to read classes from jars (#50866)

As the test classes are spread across more than one project, the Gradle
classpath contains not just folders but also jars.
This commit allows the test class to explore the archive content and
load matching classes from said source.

(cherry picked from commit 25ad74928afcbf286dc58f7d430491b0af662f04)

* QL: Remove implicit conversion inside Literal (#50962)

Literal constructor makes an implicit conversion for each value given
which turns out has some subtle side-effects.
Improve MathProcessors to preserve numeric type where possible
Fix bug on issue compatibility between date and intervals
Preserve the source when folding inside the Optimizer

(cherry picked from commit 9b73e225b0aa07a23859550fb117bae571a2b672)

* QL: Refactor DataType for pluggability (#51328)

Change DataType from enum to class
Break DataType enums into QL (default) and SQL types
Make data type conversion pluggable so that new types can be introduced

As part of the process:
- static type conversion in QL package (such as Literal) has been
removed
- several utility classes have been broken into base (QL) and extended
(SQL) parts based on type awareness
- operators (+,-,/,*) are
- due to extensibility, serialization of arithmetic operation has been
slightly changed and pushed down to the operator executor itself

(cherry picked from commit aebda81b30e1563b877a8896309fd50633e0b663)

* Compilation fixes for 7.x
2020-01-27 22:03:58 +02:00
Andrei Stefan 2908b7e5fc
SQL: add support for passing query parameters in REST API calls (#51029) (#51222)
* REST PreparedStatement-like query parameters are now supported in the form of an array of non-object, non-array values where ES SQL parser will try to infer the data type of the value being passed as parameter.

(cherry picked from commit 45b8bf619aecb1c03d7bc0cf06928dcc36005a66)
2020-01-20 16:40:19 +02:00
Andrei Stefan 543cc85b78
Add trace logging for responses coming from server (#50530) (#51221)
(cherry picked from commit 38eb485deffa175c7eb0b55a42a3e309f8a9802d)
2020-01-20 16:39:46 +02:00
Andrei Stefan df36169220
SQL: change the way unsupported data types fields are handled (#50823) (#51220)
The hierarchy of fields/sub-fields under a field that is of an
unsupported data type will be marked as unsupported as well. Until this
change, the behavior was to set the unsupported data type field's
hierarchy as empty.

Example, considering the following hierarchy of fields/sub-fields
a -> b -> c -> d, if b would be of type "foo", then b, c and d will
be marked as unsupported.

(cherry picked from commit 7adb286c4c485b9e781f88b0a2f98cab9ec5b7e2)
2020-01-20 16:23:43 +02:00
Bogdan Pintea fb65ef3f2d
SQL: Extend the optimisations for equalities (#50792) (#51098)
* Extend the optimizations for equalities

This commit supplements the optimisations of equalities in conjunctions
and disjunctions:
* for conjunctions, the existing optimizations with ranges are extended
with not-equalities and inequalities; these lead to a fast resolution,
the conjunction either being evaluate to a FALSE, or the non-equality
conditions being dropped as superfluous;
* optimisations for disjunctions are added to be applied against ranges,
inequalities and not-equalities; these lead to disjunction either
becoming TRUE or the equality being dropped, either as superfluous or
merged into a range/inequality.

* Adress review notes

* Fix the bug around wrongly optimizing 'a=2 OR a!=?', which only yields
TRUE for same values in equality and inequality.
* Var renamings, code style adjustments, comments corrections.

* Address further review comments. Extend optim.

- fix a few code comments;
- extend the Equals OR NotEquals optimitsation (a=2 OR a!=5 -> a!=5);
- extend the Equals OR Range optimisation on limits equality (a=2 OR
  2<=a<5 -> 2<=a<5);
- in case an equality is being removed in a conjunction, the rest of
  possible optimisations to test is now skipped.

* rename one var for better legiblity

- s/rmEqual/removeEquals

(cherry picked from commit 62e7c6a010f10cd7893ee5c99bad8b8d2a693436)
2020-01-16 14:32:34 +01:00
Nik Everett fc5fde7950
Add "did you mean" to ObjectParser (#50938) (#50985)
Check it out:
```
$ curl -u elastic:password -HContent-Type:application/json -XPOST localhost:9200/test/_update/foo?pretty -d'{
  "dac": {}
}'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "x_content_parse_exception",
        "reason" : "[2:3] [UpdateRequest] unknown field [dac] did you mean [doc]?"
      }
    ],
    "type" : "x_content_parse_exception",
    "reason" : "[2:3] [UpdateRequest] unknown field [dac] did you mean [doc]?"
  },
  "status" : 400
}
```

The tricky thing about implementing this is that x-content doesn't
depend on Lucene. So this works by creating an extension point for the
error message using SPI. Elasticsearch's server module provides the
"spell checking" implementation.
s
2020-01-14 17:53:41 -05:00
Bogdan Pintea f04b4cbee8
SQL: Optimisation fixes for conjunction merges (#50703) (#50933)
* SQL: Optimisation fixes for conjunction merges

This commit fixes the following issues around the way comparisions are
merged with ranges in conjunctions:
* the decision to include the equality of the lower limit is corrected;
* the selection of the upper limit is corrected to use the upper bound
of the range;
* the list of terms in the conjunction is sorted to have the ranges at
the bottom; this allows subsequent binary comarisions to find compatible
ranges and potentially be merged away. The end guarantee being that the
optimisation takes place irrespective of the order of the conjunction
terms in the statement.

Some comments are also corrected.

* adress review observation on anon. comparator

Replace anonymous comparator of split AND Expressions with a lambda.

(cherry picked from commit 9828cb143a41f1bda1219541f3a8fdc03bf6dd14)
2020-01-13 21:51:29 +01:00
Albert Zaharovits 2b789fa3e6
Make .async-search-* a restricted namespace (#50294)
Hide the `.async-search-*` in Security by making it a restricted index namespace.
The namespace is hard-coded.
To grant privileges on restricted indices, one must explicitly toggle the
`allow_restricted_indices` flag in the indices permission in the role definition.
As is the case with any other index, if a certain user lacks all permissions for an
index, that index is effectively nonexistent for that user.
2020-01-13 12:20:54 +02:00
Adrien Grand 4f2299c714
Upgrade to Lucene 8.4.0. (#50518) (#50750) 2020-01-08 18:53:59 +01:00
Adrien Grand 31158ab3d5
Add per-field metadata. (#50333)
This PR adds per-field metadata that can be set in the mappings and is later
returned by the field capabilities API. This metadata is completely opaque to
Elasticsearch but may be used by tools that index data in Elasticsearch to
communicate metadata about fields with tools that then search this data. A
typical example that has been requested in the past is the ability to attach
a unit to a numeric field.

In order to not bloat the cluster state, Elasticsearch requires that this
metadata be small:
 - keys can't be longer than 20 chars,
 - values can only be numbers or strings of no more than 50 chars - no inner
   arrays or objects,
 - the metadata can't have more than 5 keys in total.

Given that metadata is opaque to Elasticsearch, field capabilities don't try to
do anything smart when merging metadata about multiple indices, the union of
all field metadatas is returned.

Here is how the meta might look like in mappings:

```json
{
  "properties": {
    "latency": {
      "type": "long",
      "meta": {
        "unit": "ms"
      }
    }
  }
}
```

And then in the field capabilities response:

```json
{
  "latency": {
    "long": {
      "searchable": true,
      "aggreggatable": true,
      "meta": {
        "unit": [ "ms" ]
      }
    }
  }
}
```

When there are no conflicts, values are arrays of size 1, but when there are
conflicts, Elasticsearch includes all unique values in this array, without
giving ways to know which index has which metadata value:

```json
{
  "latency": {
    "long": {
      "searchable": true,
      "aggreggatable": true,
      "meta": {
        "unit": [ "ms", "ns" ]
      }
    }
  }
}
```

Closes #33267
2020-01-08 16:21:18 +01:00
Nik Everett 2362c430cd
Clean up wire test case a bit (#50627) (#50632)
* Adds JavaDoc to `AbstractWireTestCase` and
`AbstractWireSerializingTestCase` so it is more obvious you should prefer
the latter if you have a choice
* Moves the `instanceReader` method out of `AbstractWireTestCase` becaue
it is no longer used.
* Marks a bunch of methods final so it is more obvious which classes are
for what.
* Cleans up the side effects of the above.
2020-01-05 16:20:38 -05:00
Nik Everett 45663ac1a8
Use Void context on parsers where possible (#50573) (#50617)
*Most* of our parsing can be done without passing any extra context into
the parser that isn't already part of the xcontent stream. While I was
looking around at the places that *do* need a context I found a few
places that were declared to need a context but don't actually need it.
2020-01-03 13:28:55 -05:00
Marios Trivyzas f1a6b675f7 SQL: Fix issue with CAST and NULL checking. (#50371)
Previously, during expression optimisation, CAST would be considered
nullable if the casted expression resulted to a NULL literal, and would
be always non-nullable otherwise. As a result if CASE was wrapped by a
null check function like IS NULL or IS NOT NULL it was simplified to
TRUE/FALSE, eliminating the actual casting operation. So in case of an
expression with an erroneous casting like CAST('foo' AS DATETIME) IS NULL
it would be simplified to FALSE instead of throwing an Exception signifying
the attempt to cast 'foo' to a DATETIME type.

CAST now always returns Nullability.UKNOWN except from the case that
its result evaluated to a constant NULL, where it returns Nullability.TRUE.
This way the IS NULL/IS NOT NULL don't get simplified to FALSE/TRUE
and the CAST actually gets evaluated resulting to a thrown Exception.

Fixes: #50191
(cherry picked from commit 671e07a931cd828661e226cba22a5d38804a17a5)
2019-12-20 10:24:35 +02:00
Igor Motov c77ca98928 Geo: Switch generated WKT to upper case (#50285)
Switches generated WKT to upper case to
conform to the standard recommendation.

Relates #49568
2019-12-18 17:29:08 -05:00
Andrei Stefan c6fdf9ed8a Handle NULL in ResultSet's getDate() method (#50184)
(cherry picked from commit 08214eb1338fef5c8082c3f8b84c24dd53224ebe)
2019-12-17 10:03:23 +02:00
Ignacio Vera b5ec227de8
upgrade to lucene 8.4.0-snapshot-08b8d116f8f (#50129) (#50132) 2019-12-12 13:13:37 +01:00
Andrei Stefan e9e2e5fc71 Have COUNT DISTINCT return 0 instead of NULL for no documents matching. (#50037)
(cherry picked from commit cb94731e6f41bc51c23e4aab495b64eea731a061)
2019-12-12 00:34:04 +02:00
Adrien Grand 87e72156ce
Upgrade to lucene 8.4.0-snapshot-662c455. (#50016) (#50039)
Lucene 8.4 is about to be released so we should check it doesn't cause problems
with Elasticsearch.
2019-12-10 18:04:58 +01:00
Marios Trivyzas 48e7420307 SQL: [Tests] Unmute Pivot from NodeSublassTests (#49925)
The `testReplaceChildren()` has been fixed for Pivot as part
of #49693.

Reverting: #49045
(cherry picked from commit 4b9b9edbcf2041a8619b65580bbe192bf424cebc)
2019-12-09 17:20:20 +01:00
Costin Leau 5b896c5bb5
SQL: Refactor usage of NamedExpression (#49693)
To recap, Attributes form the properties of a derived table.
Each LogicalPlan has Attributes as output since each one can be part of
a query and as such its result are sent to its consumer.
This change essentially removes the name id comparison so any changes
applied to existing expressions should work as long as the said
expressions are semantically equivalent.
This change enforces the hashCode and equals which has the side-effect
of using hashCode as identifiers for each expression.
By removing any property from an Attribute, the various components need
to look the original source for comparison which, while annoying, should
prevent a reference from getting out of sync with its source due to
optimizations.

Essentially going forward there are only 3 types of NamedExpressions:

Alias - user define (implicit or explicit) name
FieldAttribute - field backed by Elasticsearch
ReferenceAttribute - a reference to another source acting as an
Attribute. Typically the Attribute of an Alias.

* Remove the usage of NamedExpression as basis for all Expressions.
Instead, restrict their use only for named context, such as projections
by using Aliasing instead.
* Remove different types of Attributes and allow only FieldAttribute,
UnresolvedAttribute and ReferenceAttribute. To avoid issues with
rewrites, resolve the references inside the QueryContainer so the
information always stays on the source.
* Side-effect, simplify the rules as the state for InnerAggs doesn't
have to be contained anymore.
* Improve ResolveMissingRef rule to handle references to named
non-singular expression tree against the same expression used up the
tree.

#49693 backport to 7.x

(cherry picked from commit 5d095e2173bcbf120f534a6f2a584185a7879b57)
2019-12-07 11:02:14 +02:00
Zachary Tong fec882a457 Decouple pipeline reductions from final agg reduction (#45796)
Historically only two things happened in the final reduction:
empty buckets were filled, and pipeline aggs were reduced (since it
was the final reduction, this was safe).  Usage of the final reduction
is growing however.  Auto-date-histo might need to perform
many reductions on final-reduce to merge down buckets, CCS
may need to side-step the final reduction if sending to a
different cluster, etc

Having pipelines generate their output in the final reduce was
convenient, but is becoming increasingly difficult to manage
as the rest of the agg framework advances.

This commit decouples pipeline aggs from the final reduction by
introducing a new "top level" reduce, which should be called
at the beginning of the reduce cycle (e.g. from the SearchPhaseController).
This will only reduce pipeline aggs on the final reduce after
the non-pipeline agg tree has been fully reduced.

By separating pipeline reduction into their own set of methods,
aggregations are free to use the final reduction for whatever
purpose without worrying about generating pipeline results
which are non-reducible
2019-12-05 16:11:54 -05:00
Andrei Stefan e2982b2110 SQL: handle NULL arithmetic operations with INTERVALs (#49633)
(cherry picked from commit ce727615c08cf5ae422feb77f69ea24fb53cd9d1)
2019-12-02 17:31:05 +02:00
Andrei Stefan 34311dd818 Fix NULL handling for FLOOR and CEIL math functions (#49644)
(cherry picked from commit 034f4cf7b4bd062c157d40f1e7a8760de31de568)
2019-12-02 17:31:04 +02:00
Andrei Stefan 4dc83a7db9 Fix Locate function optional parameter handling (#49666)
(cherry picked from commit dd3aeb8f5497bec4b050beaaf9d628a179b5454f)
2019-12-02 17:31:03 +02:00
Marios Trivyzas 901a8d1dcc
SQL: Fix issues with WEEK/ISO_WEEK/DATEDIFF (#49405)
Some extended testing with MS-SQL server and H2 (which agree on
results) revealed bugs in the implementation of WEEK related extraction
and diff functions.

Non-iso WEEK seems to be broken since #48209 because
of the replacement of Calendar and the change in the ISO rules.

ISO_WEEK failed for some edge cases around the January 1st.

DATE_DIFF was previously based on non-iso WEEK extraction which seems
not to be the case.

Fixes: #49376

(cherry picked from commit 54fe7f57289c46bb0905b1418f51a00e8c581560)
2019-11-29 17:07:30 +01:00
Mayya Sharipova 2dafecc398
Upgrade lucene to 8.4.0-snapshot-e648d601efb (#49641) 2019-11-28 11:59:58 -05:00
Marios Trivyzas b0cb7bf229 SQL: Fix issue with GROUP BY YEAR() (#49559)
Grouping By YEAR() is translated to a histogram aggregation, but
previously if there was a scalar function invloved (e.g.:
`YEAR(date + INTERVAL 2 YEARS)`), there was no proper script created
and the histogram was applied on a field with name: `date + INTERVAL 2 YEARS`
which doesn't make sense, and resulted in null result.

Check the underlying field of YEAR() and if it's a function call
`asScript()` to properly get the painless script on which the histogram
is applied.

Fixes: #49386
(cherry picked from commit 93c37abc943d00d3a14ba08435d118a6d48874c7)
2019-11-26 14:11:11 +01:00
Marios Trivyzas 3c69d4d0bd
SQL: Add TRUNC alias for TRUNCATE (#49571)
Add TRUNC as alias to already implemented TRUNCATE
numeric function which is the flavour supported by
Oracle and PostgreSQL.

Relates to: #41195

(cherry picked from commit f2aa7f0779bc5cce40cc0c1f5e5cf1a5bb7d84f0)
2019-11-26 12:32:54 +01:00
Marios Trivyzas 5d306ae3b2
SQL: Fix issue with CASE/IIF pre-calculating results (#49553)
Previously, CaseProcessor was pre-calculating (called `process()`)
on all the building elements of a CASE/IIF expression, not only the
conditions involved but also the results, as well as the final else result.
In case one of those results had an erroneous calculation
(e.g.: division by zero) this was executed and resulted in
an Exception to be thrown, even if this result was not used because of
the condition guarding it. e.g.:

```
SELECT CASE myField1 = 0 THEN NULL ELSE myField2 / myField1 END
FROM test;
```

Fixes: #49388
(cherry picked from commit dbd169afc98686cae1bc72024fad0ca32b272efd)
2019-11-26 10:48:07 +01:00
Marios Trivyzas 0c4491964b SQL: Fix issue with folding of CASE/IIF (#49449)
Add extra checks to prevent ConstantFolding rule to try to fold
the CASE/IIF functions early before the SimplifyCase rule gets applied.

Fixes: #49387

(cherry picked from commit f35c9725350e35985d8dd3001870084e1784a5ca)
2019-11-22 18:29:49 +01:00
Bogdan Pintea 8c2ab8bb72 SQL:Docs: add the PIVOT clause to SELECT section (#49129)
The PR adds the documentation on the PIVOT clause.

(cherry picked from commit a55b36065e6496c44b6e3191296931d477a8e5f5)
2019-11-20 18:21:06 +01:00
Marios Trivyzas fd1bb4a33a SQL: Fix issue with mins & hours for DATEDIFF (#49252)
Previously, DATEDIFF for minutes and hours was doing a
rounding calculation using all the time fields (secs, msecs/micros/nanos).
Instead it should first truncate the 2 dates to the respective field (mins or hours)
zeroing out all the more detailed time fields and then make the subtraction.

(cherry picked from commit 124cd18e20429e19d52fd8dc383827ea5132d428)
2019-11-19 14:25:28 +01:00
Jay Modi 57f57227ac
Clean up static web server in sql-client tests (#49187) (#49197)
The JdbcHttpClientRequestTests and HttpClientRequestTests classes both
hold a static reference to a mock web server that internally uses the
JDKs built-in HttpServer, which resides in a sun package that the
RamUsageEstimator does not have access to. This causes builds that use
a runtime of Java 8 to fail since the StaticFieldsInvariantRule is run
when Java 8 is used.

Relates #41526
Relates #49105
2019-11-15 13:02:21 -07:00
Dimitris Athanasiou be5894ed9c
[7.x][SQL] Mute JdbcConfigurationTests.testDriverConfigurationWithSSLInURL (#49085) (#49086)
Relates #41557
2019-11-14 15:15:55 +02:00
Rory Hunter c46a0e8708
Apply 2-space indent to all gradle scripts (#49071)
Backport of #48849. Update `.editorconfig` to make the Java settings the
default for all files, and then apply a 2-space indent to all `*.gradle`
files. Then reformat all the files.
2019-11-14 11:01:23 +00:00
Marios Trivyzas 7c3198ba44
SQL: [Tests] Mute testReplaceChildren for Pivot (#49045)
Temporarily "mute" the testReplaceChildren for Pivot since it leads to
failing tests for some seeds, since the new child doesn't respond to a
valid data type.

Relates to #48900

(cherry picked from commit 6200a2207b9a4264d2f3fc976577323c7e084317)
2019-11-14 11:30:33 +01:00
Mark Vieira 6ab4645f4e
[7.x] Introduce type-safe and consistent pattern for handling build globals (#48818)
This commit introduces a consistent, and type-safe manner for handling
global build parameters through out our build logic. Primarily this
replaces the existing usages of extra properties with static accessors.
It also introduces and explicit API for initialization and mutation of
any such parameters, as well as better error handling for uninitialized
or eager access of parameter values.

Closes #42042
2019-11-01 11:33:11 -07:00
Andrei Stefan e1e9b23db8 Cleanup static instance in @AfterClass 2019-10-31 23:24:40 -04:00
Andrei Stefan 2c73c7dfe3 SQL: binary communication implementation for drivers and the CLI (#48261)
* Introduce binary_format request parameter (binary.format for JDBC) to disable binary
communication between clients (jdbc/odbc) and server.
* for CLI - "binary" command line parameter (or -b) is introduced. Default value is "true".
* binary communication (cbor) is enabled by default
* disabling request parameter introduced for debugging purposes only

(cherry picked from commit f96a5ca61cb9fad9ed59357320af20e669348ce7)
2019-10-31 20:39:41 -04:00
Alexander Reelsen 4ecf234617 Upgrade to joda 2.10.4 (#47805) 2019-10-31 14:49:50 +01:00
emasab 185e067442 SQL: Failing Group By queries due to different ExpressionIds (#43072)
Fix an issue that arises from the use of ExpressionIds as keys in a lookup map
that helps the QueryTranslator to identify the grouping columns. The issue is
that the same expression in different parts of the query (SELECT clause and GROUP BY clause)
ends up with different ExpressionIds so the lookup fails. So, instead of ExpressionIds
use the hashCode() of NamedExpression.

Fixes: #41159
Fixes: #40001
Fixes: #40240
Fixes: #33361
Fixes: #46316
Fixes: #36074
Fixes: #34543
Fixes: #37044

Fixes: #42041
(cherry picked from commit 3c38ea555984fcd2c6bf9e39d0f47a01b09e7c48)
2019-10-31 14:49:16 +01:00
Mark Vieira e5c6440a4f
Simplify usage of Gradle Shadow plugin (#48478) (#48597)
This commit simplifies and standardizes our usage of the Gradle Shadow
plugin to conform more to plugin conventions. The custom "bundle" plugin
has been removed as it's not necessary and performs the same function
as the Shadow plugin's default behavior with existing configurations.

Additionally, this removes unnecessary creation of a "nodeps" artifact,
which is unnecessary because by default project dependencies will in
fact use the non-shadowed JAR unless explicitly depending on the
"shadow" configuration.

Finally, we've cleaned up the logic used for unit testing, so we are
now correctly testing against the shadow JAR when the plugin is applied.
This better represents a real-world scenario for consumers and provides
better test coverage for incorrectly declared dependencies.

(cherry picked from commit 3698131109c7e78bdd3a3340707e1c7b4740d310)
2019-10-28 12:11:55 -07:00
Marios Trivyzas 124f6d098b
SQL: [Tests] Renable CliSecurityIT (#48581)
Seems that the issue has been fixed with: #48098

Closes: #48117
(cherry picked from commit 470362361ffce794a6a12ce7a81a8029ec7d54de)
2019-10-28 15:08:38 +01:00
Przemyslaw Gomulka aaa6209be6
[7.x] [Java.time] Calculate week of a year with ISO rules BACKPORT(#48209) (#48349)
Reverting the change introducing IsoLocal.ROOT and introducing IsoCalendarDataProvider that defaults start of the week to Monday and requires minimum 4 days in first week of a year. This extension is using java SPI mechanism and defaults for Locale.ROOT only.
It require jvm property java.locale.providers to be set with SPI,COMPAT

closes #41670
backport #48209
2019-10-23 17:39:38 +02:00
Andrei Stefan 3233b59b68 Add "format" to "range" queries resulted from optimizing a logical AND (#48073)
(cherry picked from commit 020939a9bd5b34c6d540faa8b3a67b740d661be3)
2019-10-22 10:17:37 +03:00
Alpar Torok fe265f0308 Mute CliSecurityIT
tracking in #48117
2019-10-21 11:25:56 +03:00
Ignacio Vera b1224fca8c
upgrade to Lucene-8.3.0-snapshot-25968e3b75e (#48227) 2019-10-21 08:21:09 +02:00
Alpar Torok a675b35680 Mute org.elasticsearch.xpack.sql.jdbc.JdbcConfigurationTests.testDriverConfigurationWithSSLInURL
tracked in #41557
2019-10-18 11:05:56 +01:00
Marios Trivyzas 3233bce8cb
SQL: Fix issue with negative literels and parentheses (#48113)
Previously when a numeric literal was enclosed in parentheses and then
negated, the negation was lost and the number was considered positive, e.g.:
`-(5)` was considered as `5` instead of `-5`
`- ( (1.28) )` was considered as `1.28` instead of `-1.28`

Fixes: #48009

(cherry picked from commit 4dee4bf3b34081062ba2e28ab8524a066812a180)
2019-10-16 12:56:35 +02:00
Marios Trivyzas 7fddf198b7 SQL: Implement DATEDIFF function (#47920)
Implement DATEDIFF/TIMESTAMPDIFF function as per the MS-SQL spec:
https://docs.microsoft.com/en-us/sql/t-sql/functions/datediff-transact-sql?view=sql-server-2017
which allows a user to substract two date/datetime fields and return the
difference in the date/time unit specified.

Closes: #47919
(cherry picked from commit 745699f38dc8222670ffd65b66df33b5da39040b)
2019-10-15 15:12:11 +02:00
Marios Trivyzas 6589617a51
SQL: Fix arg verification for DateAddProcessor (#48041)
Previously, the safety check for the 2nd argument of the DateAddProcessor was
restricting it to Integer which was wrong since we allow all non-rational
numbers, so it's changed to a Number check as it's done in other cases.

Enhanced some tests regarding the check for an integer (non-rational
argument).

(cherry picked from commit 0516b6eaf5eb98fa5bd087c3fece80139a6b118e)
2019-10-15 12:52:11 +02:00
Marios Trivyzas 65717f6f42 SQL: Fix Nullability of DATEADD (#47921)
Previously, Nullability was set to UNKNOWN instead of TRUE which
resulted on QueryFolder not correctly folding to NULL if any of the args
was null.

Remove the overriding nullable() also for DatePart/DateTrunc to allow
delegation the parent class.

(cherry picked from commit 05a7108e133b5ae7bec2257db5ae2d30ad926ee2)
2019-10-12 13:25:08 +02:00
Marios Trivyzas 59b3294bc9 SQL: Implement DATEADD function (#47747)
Implement DATEADD/TIMESTAMPADD function as per the MS-SQL spec:
https://docs.microsoft.com/en-us/sql/t-sql/functions/dateadd-transact-sql?view=sql-server-2017
which allows a user to add/subtract specified number of specified units
to/from a date/datetime field/expression.

Closes: #47746
(cherry picked from commit e624bc281bebb4bbe0b0c2e0a8cbc712e50097a8)
2019-10-10 16:22:13 +02:00
Costin Leau dc6f0f9dc7 SQL: Re-enable muted test
Close #47080

(cherry picked from commit 63a0aa7b392f565ea01ac478fec1dd91a80202e5)
2019-10-10 15:47:47 +03:00
Marios Trivyzas c1f30e34ff SQL: Refactor binary date time functions (#47786)
Refactor DateTrunc and DatePart to use separate Pipe classes which
allows the removal of the BinaryDateOperation enum.

(cherry picked from commit a6075e7718dff94a90dbc0795dd924dcb7641092)
2019-10-10 13:52:41 +02:00
Andrei Stefan 6a4bf5de2c SQL: make date/datetime and interval types compatible in conditional functions (#47595)
(cherry picked from commit 6ff953e6396d7cc90640419aee5d036954e2eae3)
2019-10-10 13:58:35 +03:00
Andrei Stefan 75a7daae73 SQL: use calendar interval of 1y instead of fixed interval for grouping by YEAR and HISTOGRAMs (#47558)
(cherry picked from commit 55f5463eee4ecea3537df4b34645f1d87472a802)
2019-10-09 11:51:35 +03:00
Alpar Torok 36d018c909 Convert RunTask to use testclusers, remove ClusterFormationTasks (#47572)
* Convert RunTask to use testclusers, remove ClusterFormationTasks

This PR adds a new RunTask and a way for it to start a
testclusters cluster out of band and block on it to replace
the old RunTask that used ClusterFormationTasks.

With this we can now remove ClusterFormationTasks.
2019-10-08 14:43:29 +03:00
Marios Trivyzas e698e68f06 SQL: Allow whitespaces in escape patterns (#47577)
Previously, we supported only the format `{fn <FUNCTION_NAME>()}`
but other DBs like MSSQL, DB2, MariaDB/MySQL alos allow whitespaces
between `{` and `fn`. Furhermore, also some applications - like PowerBI -
generate escape sequences with spaces: `select { fn name(params) } etc.`

Add support for white spaces between `{` and the escape pattern definition
like `fn`, `ts`, `d`, `guid` etc.

Closes: #47401

(cherry picked from commit 08a22d0b393f4a76c52dabc5e7b9cafcc19c30ca)
2019-10-07 15:05:02 +02:00
Andrei Stefan a46f312ded SQL: fix multi full-text functions usage with aggregate functions (#47444)
* Skip functions involving full-text predicates when replacing multiple
aggregate functions with "stats" or "matrix_stats" aggregations.

(cherry picked from commit bb14ba83128dfb7a70f825ea08b1524072fb9ad0)
2019-10-04 16:27:22 +03:00
Alpar Torok 0a14bb174f Remove eclipse conditionals (#44075)
* Remove eclipse conditionals

We used to have some meta projects with a `-test` prefix because
historically eclipse could not distinguish between test and main
source-sets and could only use a single classpath.
This is no longer the case for the past few Eclipse versions.

This PR adds the necessary configuration to correctly categorize source
folders and libraries.
With this change eclipse can import projects, and the visibility rules
are correct e.x. auto compete doesn't offer classes from test code or
`testCompile` dependencies when editing classes in `main`.

Unfortunately the cyclic dependency detection in Eclipse doesn't seem to
take the difference between test and non test source sets into account,
but since we are checking this in Gradle anyhow, it's safe to set to
`warning` in the settings. Unfortunately there is no setting to ignore
it.

This might cause problems when building since Eclipse will probably not
know the right order to build things in so more wirk might be necesarry.
2019-10-03 11:55:00 +03:00
Henning Andersen 42453aec96 Fix XPackPlugin usages in tests (#47252)
XPackPlugin holds data in statics and can only be initialized once. This
caused tests to fail primarily when running with a low max-workers.

Replaced usages with the LocalStateCompositeXPackPlugin, which handles
this properly for testing.
2019-10-02 12:36:02 +02:00
Marios Trivyzas f792dbf239 SQL: Implement DATE_PART function (#47206)
DATE_PART(<datetime unit>, <date/datetime>) is a function that allows
the user to extract the specified unit from a date/datetime field
similar to the EXTRACT (<datetime unit> FROM <date/datetime>) but
with different names and aliases for the units and it also provides more
options like `DATE_PART('tzoffset', datetimeField)`.

Implemented following the SQL server's spec: https://docs.microsoft.com/en-us/sql/t-sql/functions/datepart-transact-sql?view=sql-server-2017
with the difference that the <datetime unit> argument is either a
literal single quoted string or gets a value from a table field, whereas
in SQL server keywords are used (unquoted identifiers) and it's not
possible to use a value coming for a table column.

Closes: #46372
(cherry picked from commit ead743d3579eb753fd314d4a58fae205e465d72e)
2019-10-01 16:28:27 +03:00
Marios Trivyzas fa0b1b641a
SQL: Add examples fo muting sql/csv integ tests (#47291)
Add examples of failures for both sql and csv integeration
tests and instructions on how to mute them.

(cherry picked from commit 591bba46516d770f5fc95a4c536dd7448b74dd49)
2019-10-01 09:12:20 +03:00
Marios Trivyzas bd2abeef40
SQL: [TESTS] Improve error messages on failures (#47308)
When an integration test fails before the assertion of the results it's
missing information, like the file name and the line in the file where
the test resides.

(cherry picked from commit 683dc7213311d13c81e06829e08f3f9f80ebf73a)
2019-09-30 22:18:39 +03:00
emasab 87156ad93b
SQL: Fix issue with duplicate columns in SELECT (#42122)
Previously, if a column (field, scalar, alias) appeared more than once in the
SELECT list, the value was returned only once (1st appearance) in each row.

Fixes: #41811

(cherry picked from commit 097ea36581a751605fc4f2088319d954ce35b5d1)
2019-09-30 15:56:29 +03:00
Marios Trivyzas 01623f9f1c
SQL: Add alias DATETRUNC to DATE_TRUNC function (#47173)
To be on the safe side in terms of use cases also add the alias
DATETRUNC to the DATE_TRUNC function.

Follows: #46473

(cherry picked from commit 9ac223cb1fc66486f86e218fa785a32b61e9bacc)
2019-09-27 15:38:51 +03:00
Costin Leau b29a2cb360 SQL: Check case where the pivot limit is reached (#47121)
In some cases, the fetch size affects the way the groups are returned
causing the last page to go beyond the limit. Add dedicated check to
prevent extra data from being returned.

Fix #47002

(cherry picked from commit f4c29646f097bbd29855300342823ef4cef61c05)
2019-09-26 22:32:42 +03:00
Igor Motov ae202fda21 SQL: Add support for shape type (#46464)
Enables support for Cartesian geometries shape type. We still need to
decide how to handle the distance function since it is currently using
the haversine distance formula and returns results in meters, which
doesn't make any sense for Cartesian geometries.

Closes #46412
Relates to #43644
2019-09-26 09:47:42 -04:00
Yannick Welsch 056ac32738 Mute JdbcCsvSpecIT.testAverageWithOneValueAndLimit
Relates to #47080
2019-09-25 10:36:53 +02:00
Dimitris Athanasiou 64bf1b56fe
[7.x] SQL: Mute pivot testAverageWithOneValueAndOrder and testSumWithoutSubquery (#47030) (#47033)
Relates #47002
2019-09-24 19:04:52 +03:00
Costin Leau a610503783 SQL: Add PIVOT support (#46489)
Add initial PIVOT support for transforming a regular table into a
statistics table around an arbitrary pivoting column:

SELECT * FROM
 (SELECT languages, country, salary, FROM mp)
 PIVOT (AVG(salary) FOR countries IN ('NL', 'DE', 'ES', 'RO', 'US'))

In the current implementation PIVOT allows only one aggregation however
this restriction is likely to be lifted in the future.
Also not all aggregations are working, in particular MatrixStats are not yet supported.

(cherry picked from commit d91263746a222915c570d4a662ec48c1d6b4f583)
2019-09-23 21:04:13 +03:00
Alpar Torok f3e67bdd17 Add resolution rule to allow resolving all deps (#46768)
Since the `resolveAllDependencies` task resolves all the congfigurations
it can find, this was not caught by our testing, but it's required to be
configuraed specifically.

We should probably cut-over to the new configurations at some point to
avoid problems like this.

Closes elastic/infra#14580
2019-09-18 11:09:43 +03:00