Commit Graph

261 Commits

Author SHA1 Message Date
Nik Everett 4d6b58495d SQL: Simplify expression ID generation (elastic/x-pack-elasticsearch#3466)
Replaces the `ExpressionIdGenerator` class with a static method and drops
the `jvmId` field from the `ExpressionId` altogether because it isn't needed.

We still have the potential to roll over expression ids if the server lives for
a long time but it is quite unlikely and rolling over only matters if we roll
over in the same query. So long as each expression id is unique to a
particular query we're fine.

Original commit: elastic/x-pack-elasticsearch@fe3d7f7216
2018-01-09 11:16:31 -05:00
Nik Everett 4009ba24b6 SQL: Document reference equality requirement
Right now `ProcessorDefinition#resolveAttributes` relies on reference
equality to detect when the rewrite does nothing. At least, all the
nodes rely on this. Maybe this isn't that important, but it is what we
do so we may as well document it.

Original commit: elastic/x-pack-elasticsearch@662372db14
2018-01-09 11:11:56 -05:00
Costin Leau 25a00a3b55 SQL: Check null in processor (elastic/x-pack-elasticsearch#3494)
Make Processors resilient to NULL values
Check null only in functions not constants

Original commit: elastic/x-pack-elasticsearch@dd8bd16d49
2018-01-09 16:18:16 +02:00
Costin Leau 2c1210610c SQL: Preserve DateTime as a type internally (elastic/x-pack-elasticsearch#3493)
Original commit: elastic/x-pack-elasticsearch@1d2f3d1dae
2018-01-08 18:46:16 +02:00
Costin Leau af33fb847e Fix the rest client
Original commit: elastic/x-pack-elasticsearch@e9aaf05398
2018-01-08 16:39:51 +02:00
Costin Leau dab50f7b77 SQL: correct index name invocation
Original commit: elastic/x-pack-elasticsearch@cec6dd6dd9
2018-01-08 14:54:18 +02:00
Costin Leau ba81321d0d SQL: Support aliases and indices pattern (elastic/x-pack-elasticsearch#3438)
Add support for aliases and indices pattern
Enhance ShowTable info to differentiate between aliases and indices
Add regex filtering of index names
Handle security exceptions (in case of no privileges or no matching)

Original commit: elastic/x-pack-elasticsearch@91e3674ca7
2018-01-06 23:08:16 +02:00
Adrien Grand 7b4178aa28 Remove redundant modifier.
Original commit: elastic/x-pack-elasticsearch@26b3539fa6
2018-01-04 18:56:36 +01:00
Adrien Grand 895246d6b1 SQL: Fix simplification of boolean expressions. (elastic/x-pack-elasticsearch#3422)
There is an error in the optimizer that causes expressions that look like
`a OR FALSE` to not be rewritten to `a`.

Original commit: elastic/x-pack-elasticsearch@8d19b77b8b
2018-01-04 15:15:09 +01:00
Igor Motov f575119a8d SQL: Add list tables and columns methods to the REST API (elastic/x-pack-elasticsearch#3464)
Adds list tables and list columns methods to the REST API. These methods are needed by JDBC and possibly other clients.

Related elastic/x-pack-elasticsearch#3419

Original commit: elastic/x-pack-elasticsearch@eaa384c7c9
2018-01-03 16:40:50 -05:00
Nik Everett a7728e5caa SQL: wrap remaining long lines (elastic/x-pack-elasticsearch#3458)
Wrap all remaining lines over 140 characters in sql for easier
readability.

Original commit: elastic/x-pack-elasticsearch@e0bc04a049
2018-01-01 13:42:38 -05:00
Nik Everett 1cc64f4ca1 SQL: Break long lines in planner package (elastic/x-pack-elasticsearch#3457)
Break lines longer than 140 characters in the planner package to make
them a bit easier to read and make checkstyle happy. It'll make some
merge conflicts but we should be able to deal with them easilly enough.

Original commit: elastic/x-pack-elasticsearch@de8c116f33
2018-01-01 11:09:28 -05:00
Nik Everett bc8103e268 SQL: Break long lines in expression package (elastic/x-pack-elasticsearch#3456)
Break lines over 140 characters in the expression package to make them a
bit easier to read and to make checkstyle happy.

Original commit: elastic/x-pack-elasticsearch@89487a79cc
2017-12-30 20:21:51 -05:00
Nik Everett c52b3350ef SQL: Break long lines in the analyzer package (elastic/x-pack-elasticsearch#3455)
Break lines longer than 140 characters in the analyzer package into
multiple lines so they are easier to read and to appease checkstyle.

Original commit: elastic/x-pack-elasticsearch@74c4c6e4ad
2017-12-30 19:09:55 -05:00
Nik Everett 16f996da92 SQL: Fix long lines in querydsl package (elastic/x-pack-elasticsearch#3454)
Break the lines longer than 140 characters in the querydsl package into
multiple lines to make them easier to read. And to make checkstyle
happy.

Original commit: elastic/x-pack-elasticsearch@cffef88490
2017-12-29 17:59:23 -05:00
Nik Everett c2527ed0f6 SQL: Break long lines in execution package (elastic/x-pack-elasticsearch#3453)
Fixes lines longer than 140 characters to be shorter.

Original commit: elastic/x-pack-elasticsearch@517ed6cde9
2017-12-29 17:05:25 -05:00
Nik Everett f7159ce56f SQL: Break long lines in parse tree walker (elastic/x-pack-elasticsearch#3451)
SQL's parse tree visitors had some lines longer than 140 characters.
This break them into multiple lines.

Original commit: elastic/x-pack-elasticsearch@58310b02f8
2017-12-29 16:52:15 -05:00
Nik Everett f5ff5a94a0 SQL: Fix long lines in function package (elastic/x-pack-elasticsearch#3452)
Adds line breaks in long lines in SQL's function package. Rewrites one
line to use `string + string` style instead of `String.format` style
because that is "more normal" in Elasticsearch.

Original commit: elastic/x-pack-elasticsearch@2f4d0358af
2017-12-29 16:52:08 -05:00
Nik Everett e45ec84a03 SQL: Fix long lines in tests (elastic/x-pack-elasticsearch#3450)
We had three tests with lines > 140 characters. This reflows those
lines.

Original commit: elastic/x-pack-elasticsearch@71b2e5fa08
2017-12-29 16:51:59 -05:00
Nik Everett 27ee786cb6 SQL: Reflow package-info.java (elastic/x-pack-elasticsearch#3448)
Reflows all the text so it wraps around 75 columns for pleasing reading.
And fixes checkstyle errors for the lines that wrap after 140!

Original commit: elastic/x-pack-elasticsearch@9630c8c32e
2017-12-29 15:26:21 -05:00
Nik Everett b99c27ba45 SQL: Collapse FunctionRegistry class hierarchy (elastic/x-pack-elasticsearch#3443)
Collapses the interface `FunctionRegistry`, and the classes
`AbstractFunctionRegistry` and `DefaultFunctionRegistry` into a single
class, `FunctionRegistry` and adds some tests for the ctor referencing
behavior inroduced in elastic/x-pack-elasticsearch#3442.

Original commit: elastic/x-pack-elasticsearch@52a697e9b8
2017-12-29 14:59:03 -05:00
Nik Everett 5dcde97fdf SQL: Fix checkstyle in sql/rule package (elastic/x-pack-elasticsearch#3446)
* Replaces long line comment with rewritten javadoc with nice line feeds
* Adds some line breaks in a few log lines

Original commit: elastic/x-pack-elasticsearch@00dc7cc9c8
2017-12-29 14:44:21 -05:00
Nik Everett 92a55df8e6 SQL: Fix checkstyle in VerificationException (elastic/x-pack-elasticsearch#3447)
Rewrite a very long line into a few shorter lines.

Original commit: elastic/x-pack-elasticsearch@72f1f98f65
2017-12-29 14:44:09 -05:00
Nik Everett 5c7395f7fc SQL: Drop unused methods from Rows (elastic/x-pack-elasticsearch#3445)
We may as well not keep unused code around. It violated checkstyle too.

Original commit: elastic/x-pack-elasticsearch@2c77b3df76
2017-12-29 14:43:50 -05:00
Nik Everett 758d16d57f SQL: Drop equals and hashCode from FunctionDefinition (elastic/x-pack-elasticsearch#3444)
They aren't used and they don't look right.

Original commit: elastic/x-pack-elasticsearch@4d07f66f0f
2017-12-29 14:43:15 -05:00
Nik Everett dbf1fc00ce SQL: Remove reflection from constructing Functions (elastic/x-pack-elasticsearch#3442)
This replaces the marker interfaces and reflection that we once used to
construct functions with method references to constructors. It uses a
few overloaded methods to build the `FunctionDefinition`s from the
method references that adapt the various forms of `Function`
constructors that we have into
`BiFunction<UnresolvedFunction, DateTimeZone, Function>` so that the
compiler can do the complex task of picking the appropriate adapter. It
is good at that sort of thing.

Many of these overloaded functions have `@SuppressWarnings("overloads")`
because they are ambiguous if you wrote one as a lambda without type
parameters. Since we always use constructor references this isn't a
problem.

It does not remove the reflection from function naming or from function
type derivation. This is big enough and these seemed significantly less
fraught.

Original commit: elastic/x-pack-elasticsearch@528d05754b
2017-12-29 12:36:09 -05:00
Nik Everett 4df10bbd78 SQL: Drop Node superclass from Query (elastic/x-pack-elasticsearch#3414)
SQL's `Node` class is a utility for performing tree traversal and
rewrites using reflection. We're not big fans of the reflection part of
that we feel that, where possible, the tree traversal should be done
explicitly.

In this case `Query`'s tree is traversed for three things:
1. To generate the Elasticsearch `QueryBuilder`s. This uses explict tree
traversal and I left it that way.
2. To ehance sorts on nested fields with the filter in the nested field.
I added `enrichNestedSort` to `Query` so each query can implement this
on its own.
3. To add inner hits that are not explicitly mentioned in the query
already when they need to be fetched. I added `containsNestedField` and
`addNestedField` to `Query` to support this.

Note: The nested field code is somewhat dead because we don't support
nested fields at the moment but SQL once did. I've tried to keep it
mostly intact but it is difficult to know for sure if it is still
functioning the same way that it has always functioned.

Original commit: elastic/x-pack-elasticsearch@5d8a8d687a
2017-12-28 17:16:55 -05:00
Igor Motov 178d6ca6b3 SQL: Move shared REST client methods to shared-client (elastic/x-pack-elasticsearch#3428)
* SQL: Move shared REST client methods to shared-client

This commit is a preliminary step for moving JDBC to the REST client. It extracts the common REST clients from CLI and moves it to shared-client. This will allow us to move to the 5 project setup: rest-proto, shared-client, server, jdbc, cli with the following dependencies:

server <-- rest-proto
shared-client <-- rest-proto
jdbc <-- shared-client
cli <-- shared-client

Relates to elastic/x-pack-elasticsearch#3419

Original commit: elastic/x-pack-elasticsearch@2e6a134de0
2017-12-28 11:10:18 -05:00
Nik Everett c6c81c940e SQL: Switch attribute resolution to OO (elastic/x-pack-elasticsearch#3433)
Switch attribute resolution from tree matching to OO. Adds
`ProcessorDefinition#resolveAttributes` which subclasess implement to
rewrite themselves against a query. `AttributeInput`s use this to
replace themselves with `ReferenceInput`s.

Original commit: elastic/x-pack-elasticsearch@97270d2ea4
2017-12-28 08:15:25 -05:00
Costin Leau f8da04dd3a SQL: don't catch Errors in the parser
Original commit: elastic/x-pack-elasticsearch@9e4324a268
2017-12-28 11:58:10 +02:00
Costin Leau cc7e11f45e SQL: document usage of parameterized type on AbstractBuilder parser (elastic/x-pack-elasticsearch#3425)
Original commit: elastic/x-pack-elasticsearch@c807f9dd40
2017-12-28 11:50:44 +02:00
Costin Leau c6760a932f SQL: Rework handling of unsupported types (elastic/x-pack-elasticsearch#3418)
Instead of ignoring or throwing an exception, unsupported types are read
If accessed, an error is returned (nicer than the mapping error).
When using * just like with object fields, they are filtered out.

Original commit: elastic/x-pack-elasticsearch@dd244f931c
2017-12-28 11:44:58 +02:00
Nik Everett e989f465bb SQL: Switch is aggs only to OO (elastic/x-pack-elasticsearch#3432)
Switches the "is this query aggs only?" question from pattern matching
on the column tree to an OO-style "ask the columns if they can be aggs
only" method.

I expect this could have been less code if I was willing to define
`supportedByAggsOnlyQuery` as `true` at the top of the
`ProcessorDefinition` object tree and override it only on nodes
`ReferenceInput` and `ScoreProcessorDefinition` but this feels dirty to
me. I tend to think of a superclass as a list of questions that all the
subclasses have to answer rather than a list of behaviors to share.
Pulling the `return true` up saves a few lines of code but breaks my
ability to reason about subclasses.


Original commit: elastic/x-pack-elasticsearch@b1338543cb
2017-12-27 14:16:22 -05:00
Costin Leau 3e7eca6f2e SQL: Improve generics usage inside Parser (elastic/x-pack-elasticsearch#3417)
Original commit: elastic/x-pack-elasticsearch@814e476b6f
2017-12-23 22:47:51 +02:00
Costin Leau 368099dcbd SQL: Add support for object/inner/dotted fields (elastic/x-pack-elasticsearch#3368)
* SQL: Add support for object/inner/dotted fields

Improve validation when selecting compound fields
Improve fallback to exact field for text fields
Consolidate various field attributes into one
Consolidate various field HitExtractors into one
Improve grammar for column declaration
Update response of embedded server
Improve index validation to allow use of internal indices
Add clean-up of scroll in case of server errors
By default thrown an exception on multi-valued fields

Original commit: elastic/x-pack-elasticsearch@45b73fe0dc
2017-12-23 13:19:54 +02:00
Igor Motov e18c7e92fa SQL: Switch CLI to REST protocol (elastic/x-pack-elasticsearch#3342)
Removes custom binary protocol used for CLI communication and switches CLI to REST.

relates elastic/x-pack-elasticsearch#3063

Original commit: elastic/x-pack-elasticsearch@b42bdcdc35
2017-12-22 20:42:13 -10:00
Nik Everett 2b11f8b3ba SQL: Preserve field order in response again
We accidentally stopped returning fields in the same order that the
user asked for them in but some docs tests caught it.

Original commit: elastic/x-pack-elasticsearch@586e3cf207
2017-12-22 17:16:15 -05:00
Costin Leau 1003cf3829 Fix formatting
Original commit: elastic/x-pack-elasticsearch@65f17636db
2017-12-22 21:09:43 +02:00
Costin Leau 2856050af9 SQL: Add javadoc and package description (elastic/x-pack-elasticsearch#3406)
SQL: Add project description as package javadoc

Original commit: elastic/x-pack-elasticsearch@106150d1e6
2017-12-22 20:29:36 +02:00
Lee Hinman 97eef004de SQL: Remove instanceof checks for field retrieval (elastic/x-pack-elasticsearch#3402)
* SQL: Remove instanceof checks for field retrieval

This removes the `instanceof` checks iterating through the columns determining
what fields need to be retrieved from the ES document. It adds an interface
`FieldExtraction` that collects the fields needed in a builder. The builder can
then build the fields necessary from a `SearchSourceBuilder`.

* Remove default implementation in favor of pushing down exception throwing

Original commit: elastic/x-pack-elasticsearch@11d3d69eb1
2017-12-21 17:11:31 -07:00
Nik Everett a288dde22f SQL: Build CLI against compact profile again (elastic/x-pack-elasticsearch#3392)
Months ago SQL used `Desktop#open` to view the results of graphviz
description of plans. We've since removed this feature in favor of just
writing the file. So now we can build the CLI against the same java
profile that we build everything agianst.

Original commit: elastic/x-pack-elasticsearch@5535f1a6dd
2017-12-20 18:49:01 -05:00
Nik Everett 1cf9d6e3f3 SQL: Document a few functions (elastic/x-pack-elasticsearch#3390)
* Starts to build the list of supported functions, adding links to
wikipedia when there is any doubt what the functions mean.
* Extracts an example of using the function from the test suite.
* Explicitly calls out how we round (half up) because there are
lots of ways to round.

Original commit: elastic/x-pack-elasticsearch@5fb64ba869
2017-12-20 17:42:29 -05:00
Lee Hinman c26f039207 SQL: Add all QUERY() query options (elastic/x-pack-elasticsearch#3389)
This adds support for (almost) all of the options that the `query_string` query
supports.

Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `QUERY()` query.

Relates to elastic/x-pack-elasticsearch#3361

Original commit: elastic/x-pack-elasticsearch@da8b29b53c
2017-12-20 15:05:40 -07:00
Lee Hinman 5b47c67dec SQL: Add all MATCH() query options (elastic/x-pack-elasticsearch#3387)
* SQL: Add all MATCH() query options

This adds support for (almost) all of the options that the `match` query
supports.

Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `MATCH()` query.

Relates to elastic/x-pack-elasticsearch#3361

* Add getters required for Node usage

Original commit: elastic/x-pack-elasticsearch@144e2bec02
2017-12-20 12:35:32 -07:00
Nik Everett 42d1c62d03 SQL: Nicer failure message one EXISTS (elastic/x-pack-elasticsearch#3384)
We don't support the SQL `EXISTS` keyword. It looks like:
```
SELECT * FROM test WHERE EXISTS (SELECT * FROM foo WHERE test.id =
foo.id)
```

It is basically a `JOIN` that doesn't return columns. Since we don't
support `JOIN`, we don't support `EXISTS`.

This PR improves the error message from "unresolved blah blah blah" to
"EXISTS is not yet supported".

relates elastic/x-pack-elasticsearch#3176

Original commit: elastic/x-pack-elasticsearch@548d57c8c1
2017-12-20 12:04:46 -05:00
Lee Hinman 38cee807b9 SQL: Parse all multi_match options from request (elastic/x-pack-elasticsearch#3361)
* Parse all multi_match options from request

Rather than hardcoding predicates that require more methods for each newly added
options, this encapsulates the state into a collection of appliers that can
accumulate what's needed in the QueryBuilder itself.

For example, the following:

```json
GET /_xpack/sql/translate
{
  "query": "SELECT foo,baz from i WHERE MATCH('baz', 'should clause', 'operator=AND;type=cross_fields')"
}
```

Then generates the following:

```json
{
  "size" : 1000,
  "query" : {
    "multi_match" : {
      "query" : "should clause",
      "fields" : [
        "baz^1.0"
      ],
      "type" : "cross_fields",
      "operator" : "AND",
      "slop" : 0,
      "prefix_length" : 0,
      "max_expansions" : 50,
      "zero_terms_query" : "NONE",
      "auto_generate_synonyms_phrase_query" : true,
      "fuzzy_transpositions" : true,
      "boost" : 1.0
    }
  },
  "_source" : {
    "includes" : [
      "baz"
    ],
    "excludes" : [ ]
  },
  "docvalue_fields" : [
    "foo"
  ]
}
```

And when an invalid field value is used:

```json
GET /_xpack/sql
{
  "query": "SELECT foo,baz from i WHERE MATCH('baz', 'should clause', 'operator=AND;minimum_should_match=potato')"
}
```

We get what ES would usually send back:

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "query_shard_exception",
        "reason" : "failed to create query: {\n  \"multi_match\" : {\n    \"query\" : \"should clause\",\n    \"fields\" : [\n      \"baz^1.0\"\n    ],\n    \"type\" : \"best_fields\",\n    \"operator\" : \"AND\",\n    \"slop\" : 0,\n    \"prefix_length\" : 0,\n    \"max_expansions\" : 50,\n    \"minimum_should_match\" : \"potato\",\n    \"zero_terms_query\" : \"NONE\",\n    \"auto_generate_synonyms_phrase_query\" : true,\n    \"fuzzy_transpositions\" : true,\n    \"boost\" : 1.0\n  }\n}",
        "index_uuid" : "ef3MWf8FTUe2Qjz2FLbhoQ",
        "index" : "i"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "i",
        "node" : "VfG9zfk9TDWdWvEZu0a4Rw",
        "reason" : {
          "type" : "query_shard_exception",
          "reason" : "failed to create query: {\n  \"multi_match\" : {\n    \"query\" : \"should clause\",\n    \"fields\" : [\n      \"baz^1.0\"\n    ],\n    \"type\" : \"best_fields\",\n    \"operator\" : \"AND\",\n    \"slop\" : 0,\n    \"prefix_length\" : 0,\n    \"max_expansions\" : 50,\n    \"minimum_should_match\" : \"potato\",\n    \"zero_terms_query\" : \"NONE\",\n    \"auto_generate_synonyms_phrase_query\" : true,\n    \"fuzzy_transpositions\" : true,\n    \"boost\" : 1.0\n  }\n}",
          "index_uuid" : "ef3MWf8FTUe2Qjz2FLbhoQ",
          "index" : "i",
          "caused_by" : {
            "type" : "number_format_exception",
            "reason" : "For input string: \"potato\""
          }
        }
      }
    ]
  },
  "status" : 400
}
```

It even includes the validation that ES already does for things like `type`:

```json
GET /_xpack/sql
{
  "query": "SELECT foo,baz from i WHERE MATCH('baz', 'should clause', 'operator=AND;type=eggplant')"
}
```

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "failed to parse [multi_match] query type [eggplant]. unknown type."
      }
    ],
    "type" : "parse_exception",
    "reason" : "failed to parse [multi_match] query type [eggplant]. unknown type."
  },
  "status" : 400
}
```

Resolves elastic/x-pack-elasticsearch#3257


Original commit: elastic/x-pack-elasticsearch@59f518af4a
2017-12-20 09:48:05 -07:00
Nik Everett 9765058e5c SQL: Remove whenClause from grammar (elastic/x-pack-elasticsearch#3378)
It wasn't used. Also remove tokens that we aren't using.

Related to elastic/x-pack-elasticsearch#3176

Original commit: elastic/x-pack-elasticsearch@3358b1e241
2017-12-20 11:06:54 -05:00
Nik Everett 2c5cfcfae4 SQL: Proper errors on set qualifiers (elastic/x-pack-elasticsearch#3377)
`SELECT DISTINCT foo FROM bar` is not yet implemented and was returning
an "unplanned item" error which isn't useful to users so I replaced it
with `SELECT DISTINCT is not yet supported`.

`SELECT foo, COUNT(*) FROM bar GROUP BY ALL foo` is not supported.
Specifically the `ALL` part. It is fairly esoteric so see [1] for what
it does. Regardless of what it does it is not widely supported and even
Microsoft's SQL Server has deprecated it so we should never support it.
Probably.

[1]: https://technet.microsoft.com/en-us/library/ms175028(v=sql.90).aspx

Related to elastic/x-pack-elasticsearch#3176

Original commit: elastic/x-pack-elasticsearch@56e5ca3009
2017-12-20 09:25:00 -05:00
Nik Everett 2cb58f3dc4 SQL: Remove ThreadLocal (elastic/x-pack-elasticsearch#3370)
`ThreadLocal` variables have a tendency to complicate control flow,
especially if these variables are "context" variables. This PR drops the
`ThreadLocal` that SQL was using for its request context in favor of
delaying construction of the components that need the context until what
they need is ready.

This further simplifies things by passing to components what they need
rather than a larger context object. This is important because not all
of the context is ready at all parts of the request. In particular we
haven't resolved the index until long after the parsing stage, but the
parser wants to know the time zone.

This way of doing things does create a few more objects on each request
but those objects are fairly light and should die as soon as the SQL has
been translated to an Elasticsearch query.

relates elastic/x-pack-elasticsearch#3178

Original commit: elastic/x-pack-elasticsearch@8331f24399
2017-12-19 12:20:23 -05:00
Nik Everett 4820bc757e SQL: Implement sorting and retrieving score (elastic/x-pack-elasticsearch#3340)
Accessing `_score` from SQL looks like:
```
--------------------------------------------------
POST /_sql
{
    "query": "SELECT SCORE(), * FROM library WHERE match(name, 'dune') ORDER BY SCORE() DESC"
}
--------------------------------------------------
```

This replaces elastic/x-pack-elasticsearch#3187

relates elastic/x-pack-elasticsearch#2887

Original commit: elastic/x-pack-elasticsearch@fe96348c22
2017-12-18 20:57:50 -05:00