This commit reenables running ITs in xpack by adding an internalClusterTest to xpack modules that contain ESIntegTestCase tests. The new task allows us to run these independently of rest integ tests, which are disabled for xpack modules because installing the bundled plugins directly is not quite the same as installing via the meta plugin. Some tests (ML) are moved to their own qa module to accommodate the need for a real cluster. A couple tests (monitoring and upgrade) have been marked as AwaitsFix.
Commits that have been folded into this commit:
* Move ML IT tests to qa/ml-native-tests
* Add internalClusterTest task and disable rest integ tests for xpack
modules. Also tweak ML tests and get upgrade tests working
* Adding the keystore and security back to the ml native tests
* Fixing native integ test
* Fix last ML test, add awaits fix to monitoring and upgrade tests
* cleanup PR
* fix checkstyle
Original commit: elastic/x-pack-elasticsearch@3c0ed6fd3b
There have been some failures in CI due to the tribe tests with security taking too long for the
cluster to form and the expected number of node to join. In the thread dumps of the failure, it
can be seen that a node is still initializing and is in a method that could take a decent amount
of time on slow machines (TokenService#computeSecretKey). This commit increases the wait time to
allow for the node to startup and join.
Original commit: elastic/x-pack-elasticsearch@2bf4c96d8f
This commit moves the source file in x-pack-core to a org.elasticsearch.xpack.core package. This is to prevent issues where we have compile-time success reaching through packages that will cross module boundaries at runtime (due to being in different classloaders). By moving these to a separate package, we have compile-time safety. Follow-ups can consider build time checking that only this package is defined in x-pack-core, or sealing x-pack-core until modules arrive for us.
Original commit: elastic/x-pack-elasticsearch@232e156e0e
Adds documentation for all of the date time functions using the new
cli-like format extracted from the csv spec. In the process of doing
this I noticed that the `WEEK` function isn't exposed as a function.
This exposes it for consistency.
Relates to elastic/x-pack-elasticsearch#2898
Original commit: elastic/x-pack-elasticsearch@0459b24cb9
This change adds SPI loading for XPackExtensions that allows to extend
XPack via an ordinary plugin. This can co-exist with the existin
extension mechanism for the time being.
Original commit: elastic/x-pack-elasticsearch@bf02b56dee
Also, removes check for whether a job-to-remove exists
and replaces it with a check of whether a job-to-remove
is already present in the calendar. This allows to
remove a job that may no longer exists and it improves
feedback for the case that an existing job is removed from
a calendar that doesn't contain it.
relates elastic/x-pack-elasticsearch#3620
Original commit: elastic/x-pack-elasticsearch@3ea39be1b6
This commit fixes compilation in audit-tests which was broken after some
stray classes were moved out of plugin into specific plugin directories.
Original commit: elastic/x-pack-elasticsearch@8bfd8cfceb
This commit moves mostly security tests that were left in plugin/src. It
also moves the CompositeTestingXPackPlugin into tribe license tests qa
project, which was the only remaining use. This class needs to be
removed, as it has unavoidable problems with dependency conflicts (ie
guava) between security and watcher, which it pulls in both into the
test classpath.
Original commit: elastic/x-pack-elasticsearch@756209e010
Introduces:
- SAML Realm
- REST & Transport actions to support SAML single signon / signout
- Tests for above
- More XML than you ever wanted to see.
Original commit: elastic/x-pack-elasticsearch@b0fe7bb652
This allows CSV tests to include a line between the header and the
values that is ignored by the test framework. This optional line can be
added to the tests to make them a little easier to read which is useful
when they are included in the documentation. As a side effect they also
closely mimick the output of the CLI. To the point where you can copy
directly from the CLI and paste into the CSV tests.
Example:
```
constantYear
// tag::year
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
year
---------------
2018
// end::year
;
```
This can be extracted with a construct like this in the docs:
```
["source","sql",subs="attributes,callouts,macros"]
--------------------------------------------------
include-tagged::{sql-specs}/datetime.csv-spec[year]
--------------------------------------------------
```
Which makes documentation that looks like this:
```
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
year
---------------
2018
```
Which is fairly nice.
Original commit: elastic/x-pack-elasticsearch@8c10b5cb10
I went to write some docs for datetime functions that look like:
```
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
year
2018
```
because I figured they'd be pretty easy to read because they didn't
require any knowledge of a data set. But it turns out that constant
folding doesn't work properly for date time functions because they don't
actually apply the extraction.
Original commit: elastic/x-pack-elasticsearch@aa9c66b2c7
Rather than run every SQL test against multiple nodes we instead run a
subset of the tests simply for speed. The subset is both small but
exercises the CLI, JDBC, and REST interface and a few special "these
might be different in against multiple nodes" cases.
This drops the run time from of `gradle clean check` of these tests from
```
BUILD SUCCESSFUL in 4m 49s
```
To:
```
BUILD SUCCESSFUL in 2m 49s
```
Original commit: elastic/x-pack-elasticsearch@dcd99bcd82
Fix show tables as JDBC with security enabled
This commit fixes the test SqlSecurityTestCase.testShowTablesWorksAsAdmin to
skip over any indices/aliases that exist starting with `.security`.
Use value comparison instead of the result sets
Fix an offset bug while at it (columns start at 1 not 0)
Resolveselastic/x-pack-elasticsearch#3423
Original commit: elastic/x-pack-elasticsearch@6fffda6070
This is the next step in removing the top level sql directory.
I named the directory `sql-cli` instead of `cli` because that puts it at
the maven coordinates `org.elasticsearch.plugin:sql-cli` instead of
`org.elasticsearch.plugin:cli`.
Relates to elastic/x-pack-elasticsearch#3363
Original commit: elastic/x-pack-elasticsearch@d41a57a136
This moves SQL's server project into `plugin:sql` without modifying how the integration is performed. I know that it is not correct with regards to the x-pack modularization but I think it is a good first step.
Original commit: elastic/x-pack-elasticsearch@2f40d02e4d
The on-closing Connection is closed so there's no leak however closing
the ResultSet is good practice and tests another piece of code as well
Original commit: elastic/x-pack-elasticsearch@3a9cee70a3
This commit adds the ability to refresh tokens that have been obtained by the API using a refresh
token. Refresh tokens are one time use tokens that are valid for 24 hours. The tokens may be used
to get a new access and refresh token if the refresh token has not been invalidated or
already refreshed.
relates elastic/x-pack-elasticsearch#2595
Original commit: elastic/x-pack-elasticsearch@23435eb815
* SQL: Wrap coercion in JdbcResultSet to throw SQLException
This catches the `ClassCastException` that could be thrown when retrieving data
from a result set, instead converting it into a `SQLException`.
Resolveselastic/x-pack-elasticsearch#3207
* Add simple test for incorrect coercion
Original commit: elastic/x-pack-elasticsearch@5480a48d95
We now separate the compiler Java home from the time runtime Java home
(the one that is used to compile class files versus the one that is used
to run tests). This commit adapts x-pack-elasticsearch to this change.
Relates elastic/x-pack-elasticsearch#3477
Original commit: elastic/x-pack-elasticsearch@bdb096e21c
* SQL: update support for nested docs
Expand sample data with nested docs
Optimize source generation for nested parents
Nested objects fields are specified in inner-hits and don't need to be
specified at the top-level query. Further more disable parent source if
not needed.
ComputingFieldRef needs to return the hitName of its wrapped children
otherwise nested values cannot be extracted
Disable GROUP BY/HAVING on nested fields
Update FieldAttribute requirements in the random tests
Original commit: elastic/x-pack-elasticsearch@e44951b5f6
This change makes the IndexLifecycleManager check for both index existence and up to date mappings
on the index prior to executing the provided runnable. Doing this provides a mechanism to make
non-breaking mapping updates to the security index in minor versions.
relates elastic/x-pack-elasticsearch#3462
Original commit: elastic/x-pack-elasticsearch@80f05d83b4
By moving tokenization for categorization to Java we give users access to considerably more options for tokenizing their log messages prior to using ML to categorize them. Now all Elasticsearch analyzer functionality is available, which opens up the possibility to sensibly categorize non-English log messages.
Relates elastic/machine-learning-cpp#491
Original commit: elastic/x-pack-elasticsearch@5d61b67614
This creates a cluster alert that triggers whenever a node is:
- Added
- Removed
- Restarted (aka Removed and Added between collections)
Unlike all previous cluster alerts, this cluster is always instantly resolved because there is no way to otherwise resolve it later (especially if nodes are replaced).
This will require a small change to the UI so that it does not simply ignore resolved alerts.
Original commit: elastic/x-pack-elasticsearch@6340bf7960
Adds a mode parameter to all SQL-related requests. The mode parameter is used for license checks as well as to define the response content. For now only two modes are supported plain (default) and jdbc. We will add other modes in the future as we add more clients.
Relates elastic/x-pack-elasticsearch#3419
Original commit: elastic/x-pack-elasticsearch@b49ca38d4b
Previously multi-line CLI SQL statements were joined, but the space command was
missing, so a command like:
```
sql> SHOW
| functions;
```
Would incorrectly parse as "showfunctions" and throw an error.
This fixes the behavior and adds a test for multi-line commands.
Resolveselastic/x-pack-elasticsearch#3410
Original commit: elastic/x-pack-elasticsearch@3870924ccd
Tweak the grammar to differentiate between table and normal identifier.
The table one allows wildcard while the normal one (for fields) does not.
Original commit: elastic/x-pack-elasticsearch@a714e950db
Add support for aliases and indices pattern
Enhance ShowTable info to differentiate between aliases and indices
Add regex filtering of index names
Handle security exceptions (in case of no privileges or no matching)
Original commit: elastic/x-pack-elasticsearch@91e3674ca7
We test on Windows with the terminal in unix mode, so we need to make sure we send `\n` instead of `\r\n`. In this case the extra `\n` after password was causing an extra new line in tests breaking them on Windows.
Original commit: elastic/x-pack-elasticsearch@721ba32bd6
* SQL: Move shared REST client methods to shared-client
This commit is a preliminary step for moving JDBC to the REST client. It extracts the common REST clients from CLI and moves it to shared-client. This will allow us to move to the 5 project setup: rest-proto, shared-client, server, jdbc, cli with the following dependencies:
server <-- rest-proto
shared-client <-- rest-proto
jdbc <-- shared-client
cli <-- shared-client
Relates to elastic/x-pack-elasticsearch#3419
Original commit: elastic/x-pack-elasticsearch@2e6a134de0
This commit hacks up the xpack repo in such a way that it moves all
actions that the transport client uses to plugin core. It also moves
any classes that those actions use to plugin core, with a few
exceptions. I tried to split up any classes that pulled in server side
logic into the client, but that was not always 100% possible. Consider
this commit a guide, and since I do not know the codebase for each
plugin, consider it a best guess for what should be moved. A few other
things were merged in below.
This commit extracts the parts of Condition and AlwaysCondition that are
needed by classes that will be moved into the client package. The only
odd thing here is that since there are two parent classes (Condition and
AlwaysCondition) it was not possible to make AlwaysConditionInteral
extend from AlwaysCondition and still parse a ConditionInternal object,
which is what the ConditionFactory expects.
This commit removes the use of internal users in the User class, and
instead moves them to Authentication, where they are used.
[insert obligatory chuck norris karate image here]
ref elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@42d0b72209
* SQL: Add support for object/inner/dotted fields
Improve validation when selecting compound fields
Improve fallback to exact field for text fields
Consolidate various field attributes into one
Consolidate various field HitExtractors into one
Improve grammar for column declaration
Update response of embedded server
Improve index validation to allow use of internal indices
Add clean-up of scroll in case of server errors
By default thrown an exception on multi-valued fields
Original commit: elastic/x-pack-elasticsearch@45b73fe0dc
* Starts to build the list of supported functions, adding links to
wikipedia when there is any doubt what the functions mean.
* Extracts an example of using the function from the test suite.
* Explicitly calls out how we round (half up) because there are
lots of ways to round.
Original commit: elastic/x-pack-elasticsearch@5fb64ba869
This adds support for (almost) all of the options that the `query_string` query
supports.
Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `QUERY()` query.
Relates to elastic/x-pack-elasticsearch#3361
Original commit: elastic/x-pack-elasticsearch@da8b29b53c
* SQL: Add all MATCH() query options
This adds support for (almost) all of the options that the `match` query
supports.
Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `MATCH()` query.
Relates to elastic/x-pack-elasticsearch#3361
* Add getters required for Node usage
Original commit: elastic/x-pack-elasticsearch@144e2bec02
We don't support the SQL `EXISTS` keyword. It looks like:
```
SELECT * FROM test WHERE EXISTS (SELECT * FROM foo WHERE test.id =
foo.id)
```
It is basically a `JOIN` that doesn't return columns. Since we don't
support `JOIN`, we don't support `EXISTS`.
This PR improves the error message from "unresolved blah blah blah" to
"EXISTS is not yet supported".
relates elastic/x-pack-elasticsearch#3176
Original commit: elastic/x-pack-elasticsearch@548d57c8c1
`SELECT DISTINCT foo FROM bar` is not yet implemented and was returning
an "unplanned item" error which isn't useful to users so I replaced it
with `SELECT DISTINCT is not yet supported`.
`SELECT foo, COUNT(*) FROM bar GROUP BY ALL foo` is not supported.
Specifically the `ALL` part. It is fairly esoteric so see [1] for what
it does. Regardless of what it does it is not widely supported and even
Microsoft's SQL Server has deprecated it so we should never support it.
Probably.
[1]: https://technet.microsoft.com/en-us/library/ms175028(v=sql.90).aspx
Related to elastic/x-pack-elasticsearch#3176
Original commit: elastic/x-pack-elasticsearch@56e5ca3009
Tests that rely on the security index and security index template being present should not remove
the template between tests as this can cause test failures. The template upgrade service relies
on cluster state updates to trigger the template being added after a delete, but there is a
scenario where the test will just wait for template that never shows up as there is no cluster
state update in that time. Instead of fighting ourselves, we should just leave the template in
place.
Relates elastic/x-pack-elasticsearch#2915
Relates elastic/x-pack-elasticsearch#2911
Original commit: elastic/x-pack-elasticsearch@3ca4aef0be
`JOIN`s aren't supported right yet so we should send back a nice 400
level error to the user telling them that.
Also pulls out some common code in `RestSqlTestCase` that I've been
staring at for a while.
Relates to elastic/x-pack-elasticsearch#3176
Original commit: elastic/x-pack-elasticsearch@1c1bd1c355
* Add remaining matrix aggregations
This adds the remaining [matrix aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/search-aggregations-matrix-stats-aggregation.html).
These aggregations aren't currently implemented due to the inter-plugin
communication not being set up, so they return "innerkey/matrix stats not
handled (yet)".
For matrix aggs that share a name with an existing aggregation (like 'count'),
they have be prefixed with "matrix_", so, "matrix_count", "matrix_mean", and
"matrix_variance".
Relates to elastic/x-pack-elasticsearch#2885
* Return HTTP 400 for innerkey/matrix stats aggs
* Add integration test for unimplemented matrix aggs
Original commit: elastic/x-pack-elasticsearch@34459c59aa
When running `gradle run` at the top level (at least with Gradle 4.4) it
attempts to run multiple instances of the server, causing the run to fail since
they can't both bind to 9200/9300.
This renames the tasks for the `qa` directories to be `runqa` and the task for
the `cli` directory to be `runcli`.
Original commit: elastic/x-pack-elasticsearch@734ab8e132
This adds:
* The CLI jar itself into the `bin`. It is an executable jar.
* A shell and bat script to start the CLI. This isn't strictly required but folks will appreciate the consistency.
* Basic packaging tests for the CLI.
Relates to elastic/x-pack-elasticsearch#2979
Original commit: elastic/x-pack-elasticsearch@158f70a530
Fixes the error message that SQL produces when it sees unsupported
indexes. It was always returning all broken indexes as "unknown" even
though we have much better error messages. It was just throwing them
away.
I caught this originally when backporting to 6.x where we had a test
that we produced a useful error message when the user attempted to run
SQL on an index with more than one type. We couldn't run that in the
feature/sql branch because it is inside the full cluster restart tests
and the "old" version of Elasticsearch used in those tests in
feature/sql is 6.x which doesn't allow indexes with multiple types. When
I backported to 6.x the test failed because it hadn't been run before.
In addition to fixing that test and the problem, this adds another test
that will reveal the problem when run in the feature/sql and master
branch.
Original commit: elastic/x-pack-elasticsearch@c7b787baee
Previously I'd added tests for JDBC and CLI that I *thought* used SSL but they didn't! I wasn't careful...
Testing changes:
* Actually enable SSL/HTTPS in the `qa:sql:security:ssl` subproject.
* Rework how `RemoteCli` handles security. This allows us to configure SSL, the keystore, and the username and password in a much less error prone way.
* Fix up JDBC tests to properly use SSL.
* Allow the `CliFixture` to specify the keystore location.
* Switch `CliFixture` and `RemoteCli` from sending the password in the connection string to filling out the prompt for it.
* Have `CliFixture` also send the keystore password when a keystore is configured.
This makes the following production code changes:
* Allow the CLI to configure the keystore location with the `-k`/`-keystore_location` parameters.
* If the keystore location is configured then the CLI will prompt for the password.
* Allow the configuration of urls starting with `https`.
* Improve the exception thrown when the URL doesn't parse by adding a suppressed exception with the original parse error, before we tried to add `http://` to the front of it.
Original commit: elastic/x-pack-elasticsearch@97fac4a3b4
This commits adds a new end point for closing in-flight cursors, it also ensures that all cursors are properly closed by adding after test checks that ensures that we don't leave any search context open.
relates elastic/x-pack-elasticsearch#2878
Original commit: elastic/x-pack-elasticsearch@1052ea28dc
* Tests: Replace YAML tests with ESTestRestCase to be able to wait for events
The YAML tests did not have any possibility to wait for the watches to
be created. A hard ten second timeout was used, that could not be
aborted, by simulating a sleep when waiting for a number of nodes that
never occured in the cluster.
This commit replaces those waiting YAML tests with ESRestTestCases, that
use `assertBusy()` to exit early once the watches have been added. Also
this increases the wait time if needed, as these tests tend to fail on
CI.
relates elastic/x-pack-elasticsearch#3217
Original commit: elastic/x-pack-elasticsearch@74b9945d88
This is mainly a promotion of Literal to Attribute to better handle folding expressions from extracted queries
Original commit: elastic/x-pack-elasticsearch@c3bb48bb61
Given that the Catalog was only ever used to hold a single index, the corresponding abstraction can be removed in favour of the abstraction that it holds, namely `GetIndexResult`.
Original commit: elastic/x-pack-elasticsearch@6932db642c
Given that we get now filtered mappings directly from the get index API (in case security is configured with FLS), we don't need the security filter nor the filtered catalog. That means we can remove the delayed action support also from AuthorizationService and rather make SQLAction a composite action like others. It will be authorized as an action, but its indices won't be checked while that will happen with its inner actions (get index and search) which need to be properly authorized.
Also, SQLGetIndicesAction is not needed anymore, as its purpose was to retrieve the indices access resolver put in the context by the security plugin for delayed actions, which are not supported anymore.
This commit kind of reverts elastic/x-pack-elasticsearch#2162, as it is now possible to integrate with security out-of-the-box
relates elastic/x-pack-elasticsearch#2934
Original commit: elastic/x-pack-elasticsearch@64d5044426
Before this was done it was easy to get into the situation where a
job created in 5.x with a default model memory limit of 4GB could not
be opened on any node in the cluster. Following this change this
problem will no longer occur for jobs that ran for a decent amount of
time on the old cluster.
relates elastic/x-pack-elasticsearch#3181
Original commit: elastic/x-pack-elasticsearch@cb029debba
* SQL: GROUP BY with multiple fields are forbidden
The check is performed in the folder Verifier as the optimizer can eliminate some fields (like those with constants)
Original commit: elastic/x-pack-elasticsearch@8d49f4ab02
SQL: Extend HAVING support
Enhance Analyzer to support HAVING scalar functions over aggregates
Enhance Analyzer to push down undeclared aggs into the Aggregate
Fix bug in Analyzer$MissingRef that caused invalid groupings to still be resolved when pushed into an Aggregate
Preserve location information across the plan
Add AttributeMap as a backing for AttributeSet
Add Optimizer rule for combining projects
Add tz to DT functions toString
Change formatTemplate to not use String.format and thus to avoid
interfering with special % chars
Extend dataset with random salary and languages
Add unit tests for AttributeMap
Fix MathFunction scripting
Improve MissingRefs to enrich UnresolvedAttribute with metadata
During the Analysis unpushed attributes are automatically enriched to
provide more accurate error information
Enhance Verifier to deal with invalid (but resolved) ORDER/HAVING
Add OrderBy arithmetic tests
Improve Verifier to prevent GROUP BY on aggregations
Add tests on grouping by scalar functions
Original commit: elastic/x-pack-elasticsearch@5030d7a755
This suite seems to be timing out on the CI slaves. Hopefully,
some extra time will stop the failures.
Original commit: elastic/x-pack-elasticsearch@95ba86be79
In order to support the source directory repo split, this commit
disables security for the regular integration tests.
The MonitoringSettingsFilterTests already existed as REST test, so
this test has been removed.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@519154dd5f
Our rolling upgrade tests were failing on many machines due to using
the 5.x default of 4GB model_memory_limit, which then propagated forward
to 6.1+ even though the default is now lower.
Original commit: elastic/x-pack-elasticsearch@3b23d8fe9d
Created a smoke-test-monitoring-with-watcher project that runs REST
tests with watcher enabled to ensure that the proper watcher are
installed either when the local or the HTTP exporter are set up.
Also removed two more watcher imports in the tests.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@0a9abc3185
SQL: Introduce PreAnalyze phase to resolve catalogs async
The new preanalyze phase collects all unresolved relations and tries
to resolve them as indices through typical async calls _before_ starting the analysis process.
The result is loaded into a catalog which is then passed to the analyzer.
While at it, the analyzer was made singleton and state across the engine
is done through SqlSession#currentContext().
Commit missing fix
Fix typo
Fix license
Fix line length
remove redundant static modifier
Remove redundant generics type
Rename catalogResolver instance member to indexResolver
Fix translate action to return a response through the listener, it hangs otherwise
IndexResolver improvements
Make sure that get index requests calls are locally executed by providing local flag.
Don't replace index/alias name with concrete index name in asCatalog response conversion. We need to preserve the original alias name for security, so it is reused in the subsequent search.
Update roles and actions names for security tests
Get index is now executed instead of sql get indices, and sql get indices has been removed.
Also made cluster privileges more restrictive to make sure that cluster state calls are no longer executed.
Fix most of the security IT tests
indices options are now unified, always lenient. The only situation where we get authorization exception back is when the user is not authorized for the sql action (besides for which indices).
Improve SessionContext handling
Fix context being invalid in non-executable phases
Make Explain & Debug command fully async
Resolve checkstyle error about redundant modifiers
Temporarily restore SqlGetIndicesAction
SqlGetIndicesAction action is still needed in RestSqlJdbcAction (metaTable and metaColumn methods), where we can't at the moment call IndexResolver directly, as security (FLS) needs index resolver to be called as part of the execution of an indices action. Once mappings are returned filtered, delayed action and the security filter will go away, as well as SqlGetIndicesAction.
SqlGetIndicesAction doesn't need to be a delayed action, my bad
[TEST] remove unused expectSqlWithAsyncLookup and rename expectSqlWithSyncLookup to expectSqlCompositeAction
Polish and feedback
Add unit test for PreAnalyzer
Original commit: elastic/x-pack-elasticsearch@57846ed613
Adds a random timezone known to both JDK and Joda as a connection parameter to all JdbcIntegrationTestCase-based tests.
Original commit: elastic/x-pack-elasticsearch@6be6a3b69b
* Use XPackRestIT as base class for XDocsClientYamlTestSuiteIT
* Remove the XPackRestTestCase class
* Address review comments
* Fix checkstyle checks
Original commit: elastic/x-pack-elasticsearch@c2a5e60c12
In order to support the repository split, this changes the
`AbstractWatcherIntegrationTestCase` to not run with security enabled.
We have a dedicated QA project called `smoke-test-watcher-with-security`,
where tests that explicitely need security should be running.
This commit removes the possibility to enable security as part of the
test case. In addition some tests have been moved over to the dedicated
project.
In addition the `timewarp` functionality cannot be configured with a
system property anymore. This would not have worked anyway, because
tests were already dependent on that functionality and did not have any
other means of running. A bit of redundant code was removed due to this.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@b24b365ad1
Drop the ssl tests against the java builtin https server. They were
failing and the failures were undebuggable. I still don't know what was
happening because you can't get any logging out of the server.
Add SSL tests against Elasticsearch because that is what actually needs
to work.
relates elastic/x-pack-elasticsearch#2870
Original commit: elastic/x-pack-elasticsearch@284cf7fb58
The /_sql endpoint now returns the results in the text format by default. Structured formats are also supported using the format parameter or accept header similar to _cat endpoints.
Original commit: elastic/x-pack-elasticsearch@4353793b83
The chained input in watcher is a useful feature to
call several endpoints before execution a condition.
However it was pretty hard to modify data from a previous
input in order to be able to execute it in another input.
This commit adds a another input, called a `transform` input,
which allows you to do a transform as another input in a chained
input.
See this example
```
"input" : {
"chain" : {
"inputs" : [ <1>
{
"first" : {
"simple" : { "path" : "/_search" }
}
},
{
"second" : {
"transform" : {
"script" : "return [ 'path' : 'ctx.payload.first.path' + '/' ]"
}
}
},
{
"third" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : 9200,
"path" : "{{ctx.payload.second.path}}" <2>
}
}
}
}
]
}
}
```
This allows for far more flexibility before executing the next input in a chained
one.
Original commit: elastic/x-pack-elasticsearch@3af9ba6e9b
In order to be able to execute a watch as the user, who stored the
watch, this commit stores certain headers of the thread context, that
was used when the watch was stored.
Upon loading the watch the headers are loaded and applied for the
following watcher execution features
* search transform
* search input
* index action
A special case is the execute watch API, which overrides the headers loaded
from the watch with the one of the current request, so that a user
cannot execute this watch with other privileges of the user who stored it.
Only the headers "es-security-runas-user", "_xpack_security_authentication" are
copied for now, as those are needed for our security features.
The headers are stored in watch status in the watch and are not returned by default,
when the GET Watch API is used. A search reveals those of course.
relates elastic/x-pack-elasticsearch#2201
Original commit: elastic/x-pack-elasticsearch@9803bd51c2
Fixes to the build system, particularly around BWC testing, and to make future
version bumps less painful.
Original commit: elastic/x-pack-elasticsearch@a1d456f30a
This change removes the InternalClient and the InternalSecurityClient. These are replaced with
usage of the ThreadContext and a transient value, `action.origin`, to indicate which component the
request came from. The security code has been updated to look for this value and ensure the
request is executed as the proper user. This work comes from elastic/x-pack-elasticsearch#2808 where @s1monw suggested
that we do this.
While working on this, I came across index template registries and rather than updating them to use
the new method, I replaced the ML one with the template upgrade framework so that we could
remove this template registry. The watcher template registry is still needed as the template must be
updated for rolling upgrades to work (see elastic/x-pack-elasticsearch#2950).
Original commit: elastic/x-pack-elasticsearch@7dbf2f263e
Adds the option to specify an elasticsearch filter in addition to the SQL query by introducing a filter parameter in the REST query which would create a boolean filter if the SQL query generates an elasticsearch query or a constant score query if SQL if the SQL query doesn't generates an elasticsearch query. Usage:
{
"query": "SELECT * FROM index",
"filter" : { "term" : { "tag" : "tech" } }
}
relates elastic/x-pack-elasticsearch#2895
Original commit: elastic/x-pack-elasticsearch@9a73813c7f
Changes to further prepare for feature split with regards to watcher:
- CryptoService has been moved into watcher
- CryptoService.generateKey() has been moved into SystemKeyTools, only
used there
- The creation of the http client/notification classes have been moved
into watcher, no further dependencies on watcher in XPackPlugin
- Each subproject now registers it's own named writeables
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@a60c98ba7e
The IDEs don't participate in the shading but gradle does. So we have to
be a little more tricky about how we set up the IDE projects, sadly.
Original commit: elastic/x-pack-elasticsearch@5196756702
* Rename REST spec xpack.deprecation.info to xpack.migration.deprecations
* Fixed parameter-type naming in xpack.ml.get_model_snapshots
* Fixed QS multi-cluster search test to use cluster.remote_info
Original commit: elastic/x-pack-elasticsearch@ccd35b4a6c
This teaches SQL to parse Elasticsearch's standard error responses
but doesn't change SQL to general Elasticsearch's standard error responses
in all cases. That can come in a followup. We do this parsing with
jackson-core, the same dependency Elasticsearch uses for parsing
json. We shade jackson-core in the JDBC driver so that users don't have to worry about
dependency clashes. We do not do so in the CLI because it is a standalone
application.
We get a few "bonus" changes along the way:
1. We save a copy operation. Before this change responses were spooled
into memory and then parsed. After this change they are parsed directly
from the response stream.
2. We had a few classes entirely to support the spooling operation that we
no longer need: `BytesArray`, `FastByteArrayInputStream`, and
`BasicByteArrayOutputStream`.
3. SQL's `Version` was incorrectly parsing the version from the jar manifest.
We didn't notice because the test was rigged to return `UNKNOWN` because
we *were* running the test from the compiled classes directory instead of the
jar. As part of shading jackson we moved running the tests to running against
the shaded jar. Now we can actually assert that we parse the version correctly.
It turns out we weren't. So I fixed it.
Original commit: elastic/x-pack-elasticsearch@2e8f397bf4
1. decouple JdbcDriver from other classes to not trigger static
initialization (this happens through JDBC service discovery)
2. reduce visibility of JdbcDriver#close so only on jar unloading it
gets triggered
3. mark 3 methods introduced in Jdbc 4.1 as unsupported (their semantics
are somewhat weird)
4. Move versioning info in one class
5. Hook Version class in both JDBC entry points to perform cp sanity
checks
6. Remove JdbcDataSource#close (DebugLog are closed when the Driver gets
unloaded by the DriverManager) as there can be multiple instances of DS
but only one for Driver known by the DriverManager
Replace Strings with constants
Properly set TZ in security tests as well
JdbcDataSource is more defensive with its internal properties
JdbcConfiguration password parameter is aligned with JDBC DriverManager
Remove usage of JdbcConnection API
Removed JdbcConnection#setTimeZone - this encourages folks to use our
private API which would tie us down.
It is somewhat limiting for folks but it has less downsides overall and
does not trip debugging (which adds a proxy unaware of this method).
Update docs
Add JdbcDataSource into the Jdbc suite
Original commit: elastic/x-pack-elasticsearch@c713665d53
This commit adds checks to the TribeWithSecurityIT tests to ensure that the security index is
writeable before making modification operations. Otherwise, we hit errors in tests that are not
always reproducible.
relates elastic/x-pack-elasticsearch#2977
Original commit: elastic/x-pack-elasticsearch@c29bdff7ae
In order to prepare for separate source directories, this commit moves
a few packages back into the watcher namespaces. A few of them have been
moved out previously as we thought that it might make sense to have a
dedicated notification API. This wont be the case for watcher on ES
anymore, so we can safely move those back into the watcher space.
Packages affected by this move:
* org.elasticsearch.xpack.common.http
* org.elasticsearch.xpack.common.text
* org.elasticsearch.xpack.common.secret
* org.elasticsearch.xpack.common.stats
* org.elasticsearch.xpack.support
* org.elasticsearch.xpack.notification
Tests have been moved accordingly.
The class `XContentUtils` has been split into one implementation for
watcher and one for security as different methods were used.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@0aec64a7e2
* Fix several NOCOMMITS
- renamed Assert to Check to make the intent clear
- clarify esMajor/Minor inside connection (thse are actually our own
methods, not part of JDBC API)
- wire pageTimeout into Cursor#nextPage
Original commit: elastic/x-pack-elasticsearch@7626c0a44a
JodaTime timezone db can be out of date compared to that of the JDK which causes the JDBC Connection to fail when the randomized tests pick a timezone that's available in the JDK but not in Joda, like SystemV/PST8. This is happening because JdbcConnection configuration is using system default timezone and tries to pass it to Elasticsearch that is using joda. This commit, explicitly sets the time zone on JdbcConnection to a time zone randomly selected from a list of timezones that are known to both JDK and Joda.
relates elastic/x-pack-elasticsearch#2812
Original commit: elastic/x-pack-elasticsearch@b02e9794a8
Ryan Ernst
Tanguy Leroux
jaymode
Costin Leau
Igor Motov
Jason Tedor
Nik Everett
Simon Willnauer
Dimitris Athanasiou
Tim Vernum
Michael Basnight
Lee Hinman
Costin Leau
Jay Modi
Lee Hinman
Alexander Reelsen
David Roberts
Chris Earle
David Kyle
javanna
David Turner
Zachary Tong
Clinton Gormley