Adapts audit logging to actions that delay getting index access control until the action is started. The audit log will contain an entry for the action itself starting without any associated indices because the indices are not yet known. The audit log will also contain an entry for every time the action resolved security for a set of indices. Since sql resolves indices one at a time it will contain an entry per index.
All of this customization is entirely in the security code. The only SQL change in this PR is to add audit logging support to the integration test.
Original commit: elastic/x-pack-elasticsearch@539bb3c2a8
The rest test waited for the watch to run in the background, but there
were no guarantees that this really happened. Also it waited for five
seconds, instead of just executing the watch manually.
relates elastic/x-pack-elasticsearch#2255
Original commit: elastic/x-pack-elasticsearch@56765a649e
These tests have repeating but not reproducible failures,
where the stash is filled with a second PUT operation and the
watcher stats response does not match. Setting the log to trace
should shed some light on this.
As the smoke tests are only four tests this will not lead to a
log explosion.
Relates elastic/x-pack-elasticsearch#1513, elastic/x-pack-elasticsearch#1874
Original commit: elastic/x-pack-elasticsearch@5832dc7990
Adds a test that shows *how* SQL fails to address an index with two types
to the full cluster restart tests. Because we're writing this code
against 7.0 don't actually execute the test, but we will execute it when
we merge to 6.x and it *should* work.
Original commit: elastic/x-pack-elasticsearch@b536e9a142
This adds support for field level security to SQL by creating a new type of flow for securing requests that look like sql requests. `AuthorizationService` verifies that the user can execute the request but doesn't check the indices in the request because they are not yet ready. Instead, it adds a `BiFunction` to the context that can be used to check permissions for an index while servicing the request. This allows requests to cooperatively secure themselves. SQL does this by implementing filtering on top of its `Catalog` abstraction and backing that filtering with security's filters. This minimizes the touch points between security and SQL.
Stuff I'd like to do in followups:
What doesn't work at all still:
1. `SHOW TABLES` is still totally unsecured
2. `DESCRIBE TABLE` is still totally unsecured
3. JDBC's metadata APIs are still totally unsecured
What kind of works but not well:
1. The audit trail doesn't show the index being authorized for SQL.
Original commit: elastic/x-pack-elasticsearch@86f88ba2f5
This commit re-enables the OpenLDAP tests that were previously running against a one-off instance
in AWS but now run against a vagrant fixture. There were some IntegTests that would run against the
OpenLDAP instance randomly but with this change they no longer run against OpenLDAP. This is ok as
the functionality that is tested by these has coverage elsewhere.
relates elastic/x-pack-elasticsearch#1823
Original commit: elastic/x-pack-elasticsearch@ac9bc82297
SQL relies on being able to fetch information about fields from
the cluster state and it'd be disasterous if that information
wasn't available. This should catch that.
Original commit: elastic/x-pack-elasticsearch@1a62747332
Running the sql rest action test inside the server caused a dependency
loop which was failing the build.
Original commit: elastic/x-pack-elasticsearch@43283671d8
Replacing integration tests with rest tests and unit tests, thus removing integration tests that require start of a new cluster. Removing unused testing methods
Original commit: elastic/x-pack-elasticsearch@265966d80c
Some standalone tests set the max script compilation limit. However,
this setting is now set in the main cluster setup in core so it is no
longer needed here. This commit removes these obviated settings in
standalone tests.
Original commit: elastic/x-pack-elasticsearch@089328c8d7
The full cluster restart tests are currently geared towards the 5.6 -> 6.0 upgrade and have some
issues when the versions are changed to 6.x -> 7.0. One issue is a real code issue in that the
security code always expects the mappings to have the same version as the version of the node, but
we no longer update the mappings on the security index during a rolling upgrade. We know look at
the index format to determine if the index is up to date.
Original commit: elastic/x-pack-elasticsearch@14c1c72ff6
The command line flag --path.conf was removed yet this test was still
using it. This commit replaces the usage of this flag in this test with
the new mechanism.
Original commit: elastic/x-pack-elasticsearch@c37050894c
Add some basic security testing/integration.
The good news:
1. Basic security now works. Users without access to an index can't run sql queries against it. Without this change they could.
2. Document level security works! At least so far as I can tell.
The work left to do:
1. Field level security doesn't work properly. I mean, it kind of works in that the field's values don't leak but it just looks like they all have null values.
2. We will need to test scrolling.
3. I've only added tests for the rest sql action. I'll need to add tests for jdbc and the CLI as well.
4. I've only added tests for `SELECT` and have ignored stuff like `DESCRIBE` and `SHOW TABLES`.
Original commit: elastic/x-pack-elasticsearch@b9909bbda0
This commit improves the error message in 6x if the security index has
not been upgraded, and warns the user that the native realm will not be
functional until the upgrade API is run.
Original commit: elastic/x-pack-elasticsearch@710b7634b4
the upgrade API is lacking some functionality in a special case,
where triggered_watches exists, but .watches does not. This
deletes the triggered watches index manually until we integrated
this properly in the upgrade API to fix the tests
Original commit: elastic/x-pack-elasticsearch@e9d1b0d35d
This commit adds the upgrade API functionality and script for security.
It also enables previously muted tests that would fail due to the lack
of security upgrade features in testing cluster restarts and old
security index backward compatibility.
Original commit: elastic/x-pack-elasticsearch@4abe9f1263
The current tests were only able to finish successfully, if the earlier
tests were run as well, you could not run the restart test in isolation.
This commit ensures an upgrade is executed if needed.
Original commit: elastic/x-pack-elasticsearch@616ebbd6eb
Otherwise we might run into race conditions that prevent a useful
start up again.
Those tests can be massively improved (no need to run against the real
master node), once the watcher BWC compatible stats are in.
relates elastic/x-pack-elasticsearch#2004
Original commit: elastic/x-pack-elasticsearch@52ca77809c
This is related to elastic/x-pack-elasticsearch#1217. This commit requires that the elastic password
be bootstrapped for the user to be authenticated. As a result it removes
the special "setup" mode that allowed the user to be authenticated from
localhost.
Additionally, this commit updates the tests to work with this
functionality.
Original commit: elastic/x-pack-elasticsearch@d0d5d697a7
When a user asks for upgrade information for all indices and there are no indices in the cluster, upgrade assistance should just return an empty response indicating that no indices require upgrade or reindexing. This commit also reverts the temporary fix in WatchBackwardsCompatibilityIT tests that was added as a workaround for this issue.
Original commit: elastic/x-pack-elasticsearch@2ea9707867
This allows for messages to be returned, and distinguishes between 4 different results:
- I have authenticated the user
- I don't know how to authenticate that user. Try another realm.
- I tried to authenticate the user, but failed. Try another realm.
- I tried to authenticate the user, but failed. Fail the authentication attempt.
Original commit: elastic/x-pack-elasticsearch@f796949cfb
When there are data or master nodes in the cluster, that are older
than ES 6.0 alpha3, then watcher will only start on the master node.
Changed all transport actions to be master node actions, as there is
already a method to decide to run locally, which we can piggyback on.
Original commit: elastic/x-pack-elasticsearch@65cecb6d69
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.
Original commit: elastic/x-pack-elasticsearch@926a7b2d86
This test creates watches in old versions of elasticsearch, upgrades them after upgrading cluster to the latest version and then tests that they were upgraded correctly.
Original commit: elastic/x-pack-elasticsearch@b9d45eb2a5
Nik Everett
Alexander Reelsen
David Roberts
Jay Modi
Ryan Ernst
Jason Tedor
David Kyle
Michael Basnight
Ali Beyad
jaymode
Tim Brooks
Igor Motov
Tim Vernum
Simon Willnauer
Jack Conradson