Previously I'd added tests for JDBC and CLI that I *thought* used SSL but they didn't! I wasn't careful...
Testing changes:
* Actually enable SSL/HTTPS in the `qa:sql:security:ssl` subproject.
* Rework how `RemoteCli` handles security. This allows us to configure SSL, the keystore, and the username and password in a much less error prone way.
* Fix up JDBC tests to properly use SSL.
* Allow the `CliFixture` to specify the keystore location.
* Switch `CliFixture` and `RemoteCli` from sending the password in the connection string to filling out the prompt for it.
* Have `CliFixture` also send the keystore password when a keystore is configured.
This makes the following production code changes:
* Allow the CLI to configure the keystore location with the `-k`/`-keystore_location` parameters.
* If the keystore location is configured then the CLI will prompt for the password.
* Allow the configuration of urls starting with `https`.
* Improve the exception thrown when the URL doesn't parse by adding a suppressed exception with the original parse error, before we tried to add `http://` to the front of it.
Original commit: elastic/x-pack-elasticsearch@97fac4a3b4
This commits adds a new end point for closing in-flight cursors, it also ensures that all cursors are properly closed by adding after test checks that ensures that we don't leave any search context open.
relates elastic/x-pack-elasticsearch#2878
Original commit: elastic/x-pack-elasticsearch@1052ea28dc
* Tests: Replace YAML tests with ESTestRestCase to be able to wait for events
The YAML tests did not have any possibility to wait for the watches to
be created. A hard ten second timeout was used, that could not be
aborted, by simulating a sleep when waiting for a number of nodes that
never occured in the cluster.
This commit replaces those waiting YAML tests with ESRestTestCases, that
use `assertBusy()` to exit early once the watches have been added. Also
this increases the wait time if needed, as these tests tend to fail on
CI.
relates elastic/x-pack-elasticsearch#3217
Original commit: elastic/x-pack-elasticsearch@74b9945d88
This is mainly a promotion of Literal to Attribute to better handle folding expressions from extracted queries
Original commit: elastic/x-pack-elasticsearch@c3bb48bb61
Given that the Catalog was only ever used to hold a single index, the corresponding abstraction can be removed in favour of the abstraction that it holds, namely `GetIndexResult`.
Original commit: elastic/x-pack-elasticsearch@6932db642c
Given that we get now filtered mappings directly from the get index API (in case security is configured with FLS), we don't need the security filter nor the filtered catalog. That means we can remove the delayed action support also from AuthorizationService and rather make SQLAction a composite action like others. It will be authorized as an action, but its indices won't be checked while that will happen with its inner actions (get index and search) which need to be properly authorized.
Also, SQLGetIndicesAction is not needed anymore, as its purpose was to retrieve the indices access resolver put in the context by the security plugin for delayed actions, which are not supported anymore.
This commit kind of reverts elastic/x-pack-elasticsearch#2162, as it is now possible to integrate with security out-of-the-box
relates elastic/x-pack-elasticsearch#2934
Original commit: elastic/x-pack-elasticsearch@64d5044426
Before this was done it was easy to get into the situation where a
job created in 5.x with a default model memory limit of 4GB could not
be opened on any node in the cluster. Following this change this
problem will no longer occur for jobs that ran for a decent amount of
time on the old cluster.
relates elastic/x-pack-elasticsearch#3181
Original commit: elastic/x-pack-elasticsearch@cb029debba
* SQL: GROUP BY with multiple fields are forbidden
The check is performed in the folder Verifier as the optimizer can eliminate some fields (like those with constants)
Original commit: elastic/x-pack-elasticsearch@8d49f4ab02
SQL: Extend HAVING support
Enhance Analyzer to support HAVING scalar functions over aggregates
Enhance Analyzer to push down undeclared aggs into the Aggregate
Fix bug in Analyzer$MissingRef that caused invalid groupings to still be resolved when pushed into an Aggregate
Preserve location information across the plan
Add AttributeMap as a backing for AttributeSet
Add Optimizer rule for combining projects
Add tz to DT functions toString
Change formatTemplate to not use String.format and thus to avoid
interfering with special % chars
Extend dataset with random salary and languages
Add unit tests for AttributeMap
Fix MathFunction scripting
Improve MissingRefs to enrich UnresolvedAttribute with metadata
During the Analysis unpushed attributes are automatically enriched to
provide more accurate error information
Enhance Verifier to deal with invalid (but resolved) ORDER/HAVING
Add OrderBy arithmetic tests
Improve Verifier to prevent GROUP BY on aggregations
Add tests on grouping by scalar functions
Original commit: elastic/x-pack-elasticsearch@5030d7a755
This suite seems to be timing out on the CI slaves. Hopefully,
some extra time will stop the failures.
Original commit: elastic/x-pack-elasticsearch@95ba86be79
In order to support the source directory repo split, this commit
disables security for the regular integration tests.
The MonitoringSettingsFilterTests already existed as REST test, so
this test has been removed.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@519154dd5f
Our rolling upgrade tests were failing on many machines due to using
the 5.x default of 4GB model_memory_limit, which then propagated forward
to 6.1+ even though the default is now lower.
Original commit: elastic/x-pack-elasticsearch@3b23d8fe9d
Created a smoke-test-monitoring-with-watcher project that runs REST
tests with watcher enabled to ensure that the proper watcher are
installed either when the local or the HTTP exporter are set up.
Also removed two more watcher imports in the tests.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@0a9abc3185
SQL: Introduce PreAnalyze phase to resolve catalogs async
The new preanalyze phase collects all unresolved relations and tries
to resolve them as indices through typical async calls _before_ starting the analysis process.
The result is loaded into a catalog which is then passed to the analyzer.
While at it, the analyzer was made singleton and state across the engine
is done through SqlSession#currentContext().
Commit missing fix
Fix typo
Fix license
Fix line length
remove redundant static modifier
Remove redundant generics type
Rename catalogResolver instance member to indexResolver
Fix translate action to return a response through the listener, it hangs otherwise
IndexResolver improvements
Make sure that get index requests calls are locally executed by providing local flag.
Don't replace index/alias name with concrete index name in asCatalog response conversion. We need to preserve the original alias name for security, so it is reused in the subsequent search.
Update roles and actions names for security tests
Get index is now executed instead of sql get indices, and sql get indices has been removed.
Also made cluster privileges more restrictive to make sure that cluster state calls are no longer executed.
Fix most of the security IT tests
indices options are now unified, always lenient. The only situation where we get authorization exception back is when the user is not authorized for the sql action (besides for which indices).
Improve SessionContext handling
Fix context being invalid in non-executable phases
Make Explain & Debug command fully async
Resolve checkstyle error about redundant modifiers
Temporarily restore SqlGetIndicesAction
SqlGetIndicesAction action is still needed in RestSqlJdbcAction (metaTable and metaColumn methods), where we can't at the moment call IndexResolver directly, as security (FLS) needs index resolver to be called as part of the execution of an indices action. Once mappings are returned filtered, delayed action and the security filter will go away, as well as SqlGetIndicesAction.
SqlGetIndicesAction doesn't need to be a delayed action, my bad
[TEST] remove unused expectSqlWithAsyncLookup and rename expectSqlWithSyncLookup to expectSqlCompositeAction
Polish and feedback
Add unit test for PreAnalyzer
Original commit: elastic/x-pack-elasticsearch@57846ed613
Adds a random timezone known to both JDK and Joda as a connection parameter to all JdbcIntegrationTestCase-based tests.
Original commit: elastic/x-pack-elasticsearch@6be6a3b69b
* Use XPackRestIT as base class for XDocsClientYamlTestSuiteIT
* Remove the XPackRestTestCase class
* Address review comments
* Fix checkstyle checks
Original commit: elastic/x-pack-elasticsearch@c2a5e60c12
In order to support the repository split, this changes the
`AbstractWatcherIntegrationTestCase` to not run with security enabled.
We have a dedicated QA project called `smoke-test-watcher-with-security`,
where tests that explicitely need security should be running.
This commit removes the possibility to enable security as part of the
test case. In addition some tests have been moved over to the dedicated
project.
In addition the `timewarp` functionality cannot be configured with a
system property anymore. This would not have worked anyway, because
tests were already dependent on that functionality and did not have any
other means of running. A bit of redundant code was removed due to this.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@b24b365ad1
Drop the ssl tests against the java builtin https server. They were
failing and the failures were undebuggable. I still don't know what was
happening because you can't get any logging out of the server.
Add SSL tests against Elasticsearch because that is what actually needs
to work.
relates elastic/x-pack-elasticsearch#2870
Original commit: elastic/x-pack-elasticsearch@284cf7fb58
The /_sql endpoint now returns the results in the text format by default. Structured formats are also supported using the format parameter or accept header similar to _cat endpoints.
Original commit: elastic/x-pack-elasticsearch@4353793b83
The chained input in watcher is a useful feature to
call several endpoints before execution a condition.
However it was pretty hard to modify data from a previous
input in order to be able to execute it in another input.
This commit adds a another input, called a `transform` input,
which allows you to do a transform as another input in a chained
input.
See this example
```
"input" : {
"chain" : {
"inputs" : [ <1>
{
"first" : {
"simple" : { "path" : "/_search" }
}
},
{
"second" : {
"transform" : {
"script" : "return [ 'path' : 'ctx.payload.first.path' + '/' ]"
}
}
},
{
"third" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : 9200,
"path" : "{{ctx.payload.second.path}}" <2>
}
}
}
}
]
}
}
```
This allows for far more flexibility before executing the next input in a chained
one.
Original commit: elastic/x-pack-elasticsearch@3af9ba6e9b
In order to be able to execute a watch as the user, who stored the
watch, this commit stores certain headers of the thread context, that
was used when the watch was stored.
Upon loading the watch the headers are loaded and applied for the
following watcher execution features
* search transform
* search input
* index action
A special case is the execute watch API, which overrides the headers loaded
from the watch with the one of the current request, so that a user
cannot execute this watch with other privileges of the user who stored it.
Only the headers "es-security-runas-user", "_xpack_security_authentication" are
copied for now, as those are needed for our security features.
The headers are stored in watch status in the watch and are not returned by default,
when the GET Watch API is used. A search reveals those of course.
relates elastic/x-pack-elasticsearch#2201
Original commit: elastic/x-pack-elasticsearch@9803bd51c2
Fixes to the build system, particularly around BWC testing, and to make future
version bumps less painful.
Original commit: elastic/x-pack-elasticsearch@a1d456f30a
This change removes the InternalClient and the InternalSecurityClient. These are replaced with
usage of the ThreadContext and a transient value, `action.origin`, to indicate which component the
request came from. The security code has been updated to look for this value and ensure the
request is executed as the proper user. This work comes from elastic/x-pack-elasticsearch#2808 where @s1monw suggested
that we do this.
While working on this, I came across index template registries and rather than updating them to use
the new method, I replaced the ML one with the template upgrade framework so that we could
remove this template registry. The watcher template registry is still needed as the template must be
updated for rolling upgrades to work (see elastic/x-pack-elasticsearch#2950).
Original commit: elastic/x-pack-elasticsearch@7dbf2f263e
Adds the option to specify an elasticsearch filter in addition to the SQL query by introducing a filter parameter in the REST query which would create a boolean filter if the SQL query generates an elasticsearch query or a constant score query if SQL if the SQL query doesn't generates an elasticsearch query. Usage:
{
"query": "SELECT * FROM index",
"filter" : { "term" : { "tag" : "tech" } }
}
relates elastic/x-pack-elasticsearch#2895
Original commit: elastic/x-pack-elasticsearch@9a73813c7f
Changes to further prepare for feature split with regards to watcher:
- CryptoService has been moved into watcher
- CryptoService.generateKey() has been moved into SystemKeyTools, only
used there
- The creation of the http client/notification classes have been moved
into watcher, no further dependencies on watcher in XPackPlugin
- Each subproject now registers it's own named writeables
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@a60c98ba7e
Nik Everett
Igor Motov
Alexander Reelsen
Costin Leau
javanna
David Roberts
Dimitrios Athanasiou
David Kyle
David Turner
Jay Modi
Zachary Tong