SQL: Introduce PreAnalyze phase to resolve catalogs async
The new preanalyze phase collects all unresolved relations and tries
to resolve them as indices through typical async calls _before_ starting the analysis process.
The result is loaded into a catalog which is then passed to the analyzer.
While at it, the analyzer was made singleton and state across the engine
is done through SqlSession#currentContext().
Commit missing fix
Fix typo
Fix license
Fix line length
remove redundant static modifier
Remove redundant generics type
Rename catalogResolver instance member to indexResolver
Fix translate action to return a response through the listener, it hangs otherwise
IndexResolver improvements
Make sure that get index requests calls are locally executed by providing local flag.
Don't replace index/alias name with concrete index name in asCatalog response conversion. We need to preserve the original alias name for security, so it is reused in the subsequent search.
Update roles and actions names for security tests
Get index is now executed instead of sql get indices, and sql get indices has been removed.
Also made cluster privileges more restrictive to make sure that cluster state calls are no longer executed.
Fix most of the security IT tests
indices options are now unified, always lenient. The only situation where we get authorization exception back is when the user is not authorized for the sql action (besides for which indices).
Improve SessionContext handling
Fix context being invalid in non-executable phases
Make Explain & Debug command fully async
Resolve checkstyle error about redundant modifiers
Temporarily restore SqlGetIndicesAction
SqlGetIndicesAction action is still needed in RestSqlJdbcAction (metaTable and metaColumn methods), where we can't at the moment call IndexResolver directly, as security (FLS) needs index resolver to be called as part of the execution of an indices action. Once mappings are returned filtered, delayed action and the security filter will go away, as well as SqlGetIndicesAction.
SqlGetIndicesAction doesn't need to be a delayed action, my bad
[TEST] remove unused expectSqlWithAsyncLookup and rename expectSqlWithSyncLookup to expectSqlCompositeAction
Polish and feedback
Add unit test for PreAnalyzer
Original commit: elastic/x-pack-elasticsearch@57846ed613
Adds a random timezone known to both JDK and Joda as a connection parameter to all JdbcIntegrationTestCase-based tests.
Original commit: elastic/x-pack-elasticsearch@6be6a3b69b
SQL: Improve grammar to better handle quotes
Fix typo in handling (back)quoted identifiers
Clarify use of unquote (dedicated for literals) and text (generic)
Address feedback
clarify that ` are picked up but not supported/recommended
Fix merge and adjust json errors to work on windows
Original commit: elastic/x-pack-elasticsearch@67e0f3f38e
The issue here is that if the first search request fails
(initScroll), then we do not have a scroll_id. However,
in order to retry the search, we reset the scroll. That
involves clearing the current search, but since we do
not have a scroll_id, the clear scroll request fails.
We end up reporting the failure for the scroll clearing,
rather than the actual problem.
This commit fixes that by avoiding clearing the scroll
when the scroll_id is null.
relates elastic/x-pack-elasticsearch#3146
Original commit: elastic/x-pack-elasticsearch@b5086028f6
This commit removes the KnownActionsTests as it no longer serves the intended purpose of catching
actions/handlers added to elasticsearch that security needs to be aware of. Today, it is common
for this test to break the build and as a mechanical response developers just add to the actions
or handlers file to get the build green again.
Relates elastic/x-pack-elasticsearch#1489
Original commit: elastic/x-pack-elasticsearch@0bdb5341f5
In order to support the repository split, this changes the
`AbstractWatcherIntegrationTestCase` to not run with security enabled.
We have a dedicated QA project called `smoke-test-watcher-with-security`,
where tests that explicitely need security should be running.
This commit removes the possibility to enable security as part of the
test case. In addition some tests have been moved over to the dedicated
project.
In addition the `timewarp` functionality cannot be configured with a
system property anymore. This would not have worked anyway, because
tests were already dependent on that functionality and did not have any
other means of running. A bit of redundant code was removed due to this.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@b24b365ad1
This change applies the same pattern that was applied in elastic/x-pack-elasticsearch#3054 to the
ML batched results iterators, which are used to scroll through ML results
during some internal ML implementation details, such as renormalization
and nightly maintenance.
Additionally the thread context is reset before submitting the results
processor to a thread pool, to avoid masking the problem in situations
where the user opening the job coincidentally had workable permissions.
Fixeselastic/machine-learning-cpp#438
Original commit: elastic/x-pack-elasticsearch@bd1e2dc7d4
While we're fairly sure we're going to remove the binary protocol in the
long run, we're also fairly sure we're going to release the first
version of SQL with the binary protocol. One big problem with it is that
it blows up when it attempts to serialize fairly long strings. These
long strings are actually quite common in the CLI. They are also
possible in JDBC. I say "fairly long strings" because exactly how long
the strings has to be is kind of funky. It is based on the number of
bytes that it takes to encode the string, and the strings are encoded in
a utf-8-like encoding of utf-16 encoded string documented here:
https://docs.oracle.com/javase/7/docs/api/java/io/DataOutput.html#writeUTF(java.lang.String)
Anyway, this fixes the protocol for these "fairly long strings" by
chunking the strings and adding an extra 4 byte integer before each
string to count the number of chunks. After that 4 byte integer the
strings are serialized using the "normal" DataInput/DataOutput encoding,
the funny utf-8-like encoding of the utf-16 encoded string.
relates elastic/x-pack-elasticsearch#3018
Original commit: elastic/x-pack-elasticsearch@11f0d59f20
Drop the ssl tests against the java builtin https server. They were
failing and the failures were undebuggable. I still don't know what was
happening because you can't get any logging out of the server.
Add SSL tests against Elasticsearch because that is what actually needs
to work.
relates elastic/x-pack-elasticsearch#2870
Original commit: elastic/x-pack-elasticsearch@284cf7fb58
The /_sql endpoint now returns the results in the text format by default. Structured formats are also supported using the format parameter or accept header similar to _cat endpoints.
Original commit: elastic/x-pack-elasticsearch@4353793b83
* [DOCS] Move migrate tool reference
* [DOCS] Fixed link to migration tool
* [DOCS] Small edits to the migrate tool parameters
* [DOCS] Fixed migrate tool example
Original commit: elastic/x-pack-elasticsearch@0ff40ebdcc
* [DOCS] Enabled code snippet testing for start datafeed API
* [DOCS] Added datafeed creation to build.gradle
Original commit: elastic/x-pack-elasticsearch@1acb452cf0
The chained input in watcher is a useful feature to
call several endpoints before execution a condition.
However it was pretty hard to modify data from a previous
input in order to be able to execute it in another input.
This commit adds a another input, called a `transform` input,
which allows you to do a transform as another input in a chained
input.
See this example
```
"input" : {
"chain" : {
"inputs" : [ <1>
{
"first" : {
"simple" : { "path" : "/_search" }
}
},
{
"second" : {
"transform" : {
"script" : "return [ 'path' : 'ctx.payload.first.path' + '/' ]"
}
}
},
{
"third" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : 9200,
"path" : "{{ctx.payload.second.path}}" <2>
}
}
}
}
]
}
}
```
This allows for far more flexibility before executing the next input in a chained
one.
Original commit: elastic/x-pack-elasticsearch@3af9ba6e9b
We have to ensure logging is configured for any CLI command that depends
on core Elasticsearch (since it might directly or indirectly touch
logging). This commit does this for all commands in X-Pack.
Relates elastic/x-pack-elasticsearch#3112
Original commit: elastic/x-pack-elasticsearch@f77f9b5052
In order to be able to execute a watch as the user, who stored the
watch, this commit stores certain headers of the thread context, that
was used when the watch was stored.
Upon loading the watch the headers are loaded and applied for the
following watcher execution features
* search transform
* search input
* index action
A special case is the execute watch API, which overrides the headers loaded
from the watch with the one of the current request, so that a user
cannot execute this watch with other privileges of the user who stored it.
Only the headers "es-security-runas-user", "_xpack_security_authentication" are
copied for now, as those are needed for our security features.
The headers are stored in watch status in the watch and are not returned by default,
when the GET Watch API is used. A search reveals those of course.
relates elastic/x-pack-elasticsearch#2201
Original commit: elastic/x-pack-elasticsearch@9803bd51c2
Fixes to the build system, particularly around BWC testing, and to make future
version bumps less painful.
Original commit: elastic/x-pack-elasticsearch@a1d456f30a
* [DOCS] Added ML forecast API
* [DOCS] Added forecast API to build.gradle
* [DOCS] Added forecast API example
* [DOCS] Fixed forecast API intro
* [DOCS] Addressed feedback on forecast API
* [DOCS] Added duration to forecast API
* [DOCS] Removed end time from forecast API
* [DOCS] Fixed gradle errors for forecast API
Original commit: elastic/x-pack-elasticsearch@db79e3d5bb
The 6.x and 6.0 versions of the documentation show
```yml
transport.profiles.client.xpack.security.ssl.client_authentication: no
```
Which results in
```
2017-11-22T11:13:33,225][ERROR][org.elasticsearch.bootstrap.Bootstrap] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:452) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:392) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:142) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.<init>(Node.java:302) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.0.0.jar:6.0.0]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443) ~[elasticsearch-6.0.0.jar:6.0.0]
... 14 more
Caused by: java.lang.IllegalArgumentException: could not resolve ssl client auth. unknown value [no]
at org.elasticsearch.xpack.ssl.SSLClientAuth.parse(SSLClientAuth.java:78) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLConfigurationSettings.lambda$null$27(SSLConfigurationSettings.java:183) ~[?:?]
at org.elasticsearch.common.settings.Setting.get(Setting.java:352) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.settings.Setting.get(Setting.java:346) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.xpack.ssl.SSLConfiguration.<init>(SSLConfiguration.java:80) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.lambda$loadSSLConfigurations$1(SSLService.java:462) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_144]
at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:461) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.<init>(SSLService.java:87) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:237) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443) ~[elasticsearch-6.0.0.jar:6.0.0]
... 14 more
```
I think the valid value there is `none`, so
```yml
transport.profiles.client.xpack.security.ssl.client_authentication: none
```
The tests seem to confirm that https://github.com/elastic/x-pack-elasticsearch/blob/elastic/x-pack-elasticsearch@4860e92d906e046a23aa07b39ee6ef637f011dc1/plugin/src/test/java/org/elasticsearch/xpack/ssl/SSLServiceTests.java#L269
Original commit: elastic/x-pack-elasticsearch@a35b3ac8c9
Instead of returning "error response" objects and then translating them
into SQL exceptions this just throws the SQL exceptions directly. This
means the CLI catches exceptions and prints out the messages which isn't
ideal if this were hot code but it isn't and this is a much simpler way
of doing things.
Original commit: elastic/x-pack-elasticsearch@08431d3941
Costin Leau
Igor Motov
Costin Leau
Nik Everett
Dimitris Athanasiou
Lisa Cawley
David Kyle
Jay Modi
Michael Basnight
Adrien Grand
Alexander Reelsen
David Roberts
Jason Tedor
Russ Cam
David Turner
jaymode
Igor Kupczyński