Commit Graph

14 Commits

Author SHA1 Message Date
Costin Leau 368099dcbd SQL: Add support for object/inner/dotted fields (elastic/x-pack-elasticsearch#3368)
* SQL: Add support for object/inner/dotted fields

Improve validation when selecting compound fields
Improve fallback to exact field for text fields
Consolidate various field attributes into one
Consolidate various field HitExtractors into one
Improve grammar for column declaration
Update response of embedded server
Improve index validation to allow use of internal indices
Add clean-up of scroll in case of server errors
By default thrown an exception on multi-valued fields

Original commit: elastic/x-pack-elasticsearch@45b73fe0dc
2017-12-23 13:19:54 +02:00
Nik Everett 4820bc757e SQL: Implement sorting and retrieving score (elastic/x-pack-elasticsearch#3340)
Accessing `_score` from SQL looks like:
```
--------------------------------------------------
POST /_sql
{
    "query": "SELECT SCORE(), * FROM library WHERE match(name, 'dune') ORDER BY SCORE() DESC"
}
--------------------------------------------------
```

This replaces elastic/x-pack-elasticsearch#3187

relates elastic/x-pack-elasticsearch#2887

Original commit: elastic/x-pack-elasticsearch@fe96348c22
2017-12-18 20:57:50 -05:00
Lee Hinman 632c3e8238 SQL: Fix running `gradle run` from top-level directory (elastic/x-pack-elasticsearch#3329)
When running `gradle run` at the top level (at least with Gradle 4.4) it
attempts to run multiple instances of the server, causing the run to fail since
they can't both bind to 9200/9300.

This renames the tasks for the `qa` directories to be `runqa` and the task for
the `cli` directory to be `runcli`.

Original commit: elastic/x-pack-elasticsearch@734ab8e132
2017-12-15 08:42:18 -07:00
Costin Leau cdfe0f1f5a Revert "Revert "Merge branch 'feature/sql'""
This reverts commit elastic/x-pack-elasticsearch@cc79e19911.

We'll merge this when we're good and ready.

Original commit: elastic/x-pack-elasticsearch@b3ef4f2836
2017-12-13 10:19:31 -05:00
Costin Leau 2e60e831c0 Revert "Merge branch 'feature/sql'"
This reverts commit elastic/x-pack-elasticsearch@2b3f7697a5, reversing
changes made to elastic/x-pack-elasticsearch@b79f16673c.

We're backing out all of SQL.

Original commit: elastic/x-pack-elasticsearch@cc79e19911
2017-12-13 09:33:13 -05:00
Igor Motov 6839f99ed0 SQL: Switch CLI to core-cli's Command and make it testable (elastic/x-pack-elasticsearch#3232)
Switches CLI to use the standard Elasticsearch Command and refactors it to be more testable. It doesn't change any cli functionality except using the bright color while displaying query results.

relates elastic/x-pack-elasticsearch#2881, elastic/x-pack-elasticsearch#3203, elastic/x-pack-elasticsearch#2990

Original commit: elastic/x-pack-elasticsearch@841f306d50
2017-12-06 11:27:51 -05:00
Costin Leau 4e49769efb SQL: Extend HAVING support (elastic/x-pack-elasticsearch#3155)
SQL: Extend HAVING support

Enhance Analyzer to support HAVING scalar functions over aggregates
Enhance Analyzer to push down undeclared aggs into the Aggregate
Fix bug in Analyzer$MissingRef that caused invalid groupings to still be resolved when pushed into an Aggregate
Preserve location information across the plan
Add AttributeMap as a backing for AttributeSet
Add Optimizer rule for combining projects
Add tz to DT functions toString
Change formatTemplate to not use String.format and thus to avoid
interfering with special % chars
Extend dataset with random salary and languages
Add unit tests for AttributeMap
Fix MathFunction scripting
Improve MissingRefs to enrich UnresolvedAttribute with metadata
During the Analysis unpushed attributes are automatically enriched to
provide more accurate error information
Enhance Verifier to deal with invalid (but resolved) ORDER/HAVING
Add OrderBy arithmetic tests
Improve Verifier to prevent GROUP BY on aggregations
Add tests on grouping by scalar functions

Original commit: elastic/x-pack-elasticsearch@5030d7a755
2017-12-05 18:14:15 +02:00
Costin Leau 7cab29760d SQL: Introduce PreAnalyze phase to resolve catalogs async (elastic/x-pack-elasticsearch#2962)
SQL: Introduce PreAnalyze phase to resolve catalogs async

The new preanalyze phase collects all unresolved relations and tries
to resolve them as indices through typical async calls _before_ starting the analysis process.
The result is loaded into a catalog which is then passed to the analyzer.
While at it, the analyzer was made singleton and state across the engine
is done through SqlSession#currentContext().


Commit missing fix
Fix typo
Fix license
Fix line length
remove redundant static modifier
Remove redundant generics type
Rename catalogResolver instance member to indexResolver
Fix translate action to return a response through the listener, it hangs otherwise
IndexResolver improvements

Make sure that get index requests calls are locally executed by providing local flag.
Don't replace index/alias name with concrete index name in asCatalog response conversion. We need to preserve the original alias name for security, so it is reused in the subsequent search.
Update roles and actions names for security tests
Get index is now executed instead of sql get indices, and sql get indices has been removed.
Also made cluster privileges more restrictive to make sure that cluster state calls are no longer executed.
Fix most of the security IT tests
indices options are now unified, always lenient. The only situation where we get authorization exception back is when the user is not authorized for the sql action (besides for which indices).
Improve SessionContext handling
Fix context being invalid in non-executable phases
Make Explain & Debug command fully async
Resolve checkstyle error about redundant modifiers
Temporarily restore SqlGetIndicesAction

SqlGetIndicesAction action is still needed in RestSqlJdbcAction (metaTable and metaColumn methods), where we can't at the moment call IndexResolver directly, as security (FLS) needs index resolver to be called as part of the execution of an indices action. Once mappings are returned filtered, delayed action and the security filter will go away, as well as SqlGetIndicesAction.
SqlGetIndicesAction doesn't need to be a delayed action, my bad

[TEST] remove unused expectSqlWithAsyncLookup and rename expectSqlWithSyncLookup to expectSqlCompositeAction
Polish and feedback
Add unit test for PreAnalyzer

Original commit: elastic/x-pack-elasticsearch@57846ed613
2017-11-30 18:18:08 +02:00
Nik Everett 33f4a8317c JDBC metadata integration with security (elastic/x-pack-elasticsearch#2806)
I realized that we weren't running our DatabaseMetaData tests. One thing led to another and I made these changes:
1. Got the DatabaseMetaData tests running in all three of our QA projects.
2. Fixed the SecurityCatalogFilter to work with `SqlGetIndicesAction`. It worked before, but only for requests that were a `SqlAction` as well as `SqlGetIndicesAction`.
3. Added security test for the JDBC DatabaseMetaData requests. These mirror exactly the security tests that we use for `SHOW TABLES` and `DESCRIBE` but cover the JDBC actions.

Original commit: elastic/x-pack-elasticsearch@7026d83c06
2017-10-30 23:22:12 +00:00
Nik Everett d933b1b48b Initial docs for SQL (elastic/x-pack-elasticsearch#2810)
Adds docs for the REST API, translate API, the CLI, and JDBC.

Next we need to add more example queries and documentation for our
extensions.

Original commit: elastic/x-pack-elasticsearch@ed6d1360d2
2017-10-30 17:23:27 +00:00
Nik Everett 852af7de57 Fix error handling in SQL's CLI (elastic/x-pack-elasticsearch#2730)
We weren't returning errors correctly from the server
or catching them correctly in the CLI. This fixes that
and adds simple integration tests.

Original commit: elastic/x-pack-elasticsearch@259da0da6f
2017-10-12 16:32:15 +00:00
Nik Everett c7c79bc1c0 Add scrolling support to jdbc (elastic/x-pack-elasticsearch#2524)
* Switch `ResultSet#getFetchSize` from returning the *requested*
fetch size to returning the size of the current page of results.
For scrolling searches without parent/child this'll match the
requested fetch size but for other things it won't. The nice thing
about this is that it lets us tell users a thing to look at if
they are wondering why they are using a bunch of memory.
* Remove all the entire JDBC action and implement it on the REST
layer instead.
* Create some code that can be shared between the cli and jdbc
actions to properly handle errors and request deserialization so
there isn't as much copy and paste between the two. This helps
because it calls out explicitly the places where they are different.
  * I have not moved the CLI REST handle to shared code because
I think it'd be clearer to make that change in a followup PR.
* There is now no more need for constructs that allow iterating
all of the results in the same request. I have not removed these
because I feel that that cleanup is big enough to deserve its own
PR.

Original commit: elastic/x-pack-elasticsearch@3b12afd11c
2017-09-25 14:41:46 -04:00
Nik Everett 8a05c1b81f Move all sql integration tests into qa (elastic/x-pack-elasticsearch#2432)
Builds on elastic/x-pack-elasticsearch#2403 to move all of sql's integration testing into
qa modules with different running server configurations. The
big advantage of this is that it allows us to test the cli and
jdbc with security present.

Creating a project that depends on both cli and jdbc and the
server has some prickly jar hell issues because cli and jdbc
package their dependencies in the jar. This works around it
in a few days:
1. Include only a single copy of the JDBC dependencies with
careful gradle work.
2. Do not include the CLI on the classpath at all and instead
run it externally.

I say "run it externally" rather than "fork it" because Elasticsearch
tests aren't allowed to fork other processes. This is forbidden
by seccomp on linux and seatbelt on osx and cannot be explicitly
requested like additional security manager settings. So instead
of forking the CLI process directly the tests interact with a test
fixture that isn't bound by Elasticsearch's rules and *can* fork
it.

This forking of the CLI has a nice side effect: it forces us to
make sure that things like security and connection strings other
than `localhost:9200` work. The old test could and did work around
missing features like that. The new tests cannot so I added the
ability to set the connection string. Configuring usernames and
passwords was also not supported but I did not add support for
that, only created the failing test and marked it as `@AwaitsFix`.

Original commit: elastic/x-pack-elasticsearch@560c6815e3
2017-09-21 09:58:52 -04:00
Nik Everett 97ddee8d1b Shuffle SQL's integration tests some (elastic/x-pack-elasticsearch#2403)
This shuffles all of SQL's QA tests into the `qa/sql` directory, moving
some shared resources into the new `qa:sql` project. It also rigs up
testing of the rest SQL interface in all the sql qa configurations:
without security, with security, and against multiple nodes.

I've had to make some modifications to how we handle the audit log
because it has gotten pretty slow. If these modifications turn out to
not be fast enough then I'll change the test to querying the log files
and drop the audit log index entirely but the index seems to be holding
out for now.

Original commit: elastic/x-pack-elasticsearch@ff3b5a74c1
2017-08-30 17:22:46 -04:00