This commit removes the FAILED state for the IndexAuditTrail so that we always try to keep starting
the service. Previously, on any exception during startup we moved to a failed state and never tried
to start again. The users only option was to restart the node. This was problematic in the case of
large clusters as there could be common timeouts of cluster state listeners that would cause the
startup of this service to fail.
Additionally, the logic in the IndexAuditTrail to update the template on the current cluster has
been removed and replaced with the use of the TemplateUpgradeService. However, we still need to
maintain the ability to determine if a template on a remote cluster should be PUT. To avoid always
PUTing the template, the version field has been added so it only needs to be PUT once on upgrade.
Finally, the default queue size has been increased as this is another common issue that users hit
with high traffic clusters.
relates elastic/x-pack-elasticsearch#2658
Original commit: elastic/x-pack-elasticsearch@27e2ce7223
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.
This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.
The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.
Original commit: elastic/x-pack-elasticsearch@981f7a40e5
This commit replaces the REST test that the global checkpoint sync
action runs successfully as a privileged user. The test needs to be
replaced because it has a small race condition. Namely, the check that
the post-operation global checkpoint sync was successful could run
before the sync finishes running. To address this, we replace the REST
test with a test where we have a little more control and can assert busy
to avoid this race from failing the test.
Relates elastic/x-pack-elasticsearch#2749
Original commit: elastic/x-pack-elasticsearch@ea585b843c
After a write operation on an index, a post-operation global checkpoint
sync fires. Previously, this action fired on the same user as executed
the write action. If the user did not have priviledges for this action,
the global checkpoint sync would fail. With an upstream change in core,
this action now fires as the system user. This commit adds a test that
create a user that has minimal write permissions on an index, but none
that would imply it could execute the global checkpoint sync. This then
serves as a test that the upstream change to fire the global checkpoint
sync as the system user is correct. This test must run as a mulit-node
test so that a replica is a assigned so that the global checkpoint sync
fires in the first place. This test does indeed fail without the
upstream change, and passes with it.
Relates elastic/x-pack-elasticsearch#2744
Original commit: elastic/x-pack-elasticsearch@bf7e771756
This commit fixes indentation in certgen.bash, adds a check on cluster
health in bootstrap_password.bash and fixes a bug in xpack.bash
Original commit: elastic/x-pack-elasticsearch@d6847f6640
Adds the GET overall_buckets API.
The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets
The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.
relates elastic/x-pack-elasticsearch#2693
Original commit: elastic/x-pack-elasticsearch@ba6061482d
Do not execute bind on on the LDAP reader thread
Each LDAP connection has a single associated thread, executing the handlers for async requests; this is managed by the LDAP library. The bind operation is blocking for the connection. It is a deadlock to call bind, if on the LDAP reader thread for the same connection, because waiting for the bind response blocks the thread processing responses (for this connection).
This will execute the bind operation (and the subsequent runnable) on a thread pool after checking for the conflict above.
Closes: elastic/x-pack-elasticsearch#2570, elastic/x-pack-elasticsearch#2620
Original commit: elastic/x-pack-elasticsearch@404a3d8737
Since elastic/elasticsearch#26878, array and list of settings are
internally represented as actual lists. This makes filtering works
as expected when it comes to filter out arrays/lists.
The packaging tests used to check the presence of the XPack SSL
certificated_authorities setting which should have always been filtered.
By fixing the filtering of settings, elastic/elasticsearch#26878 broke
this packaging test.
This commit changes this test so that it does not expect certificated_authorities
setting to exist in the Nodes Info response.
relates elastic/x-pack-elasticsearch#2688
Original commit: elastic/x-pack-elasticsearch@cb299186b8
The true purpose of this test is to introduce another test alongside
the original, so that the test suite passes even if the other test
is skipped due to the assumption it makes about `build.snapshot`.
Original commit: elastic/x-pack-elasticsearch@709d7a5dc5
Release tests were introduced that sets the `build.snapshot`
system property to `false` to mimic release builds. This invalidates
the hardcoded license signatures that were signed against the
integration test pub/priv keys. This commit modifies the
license-validation assertions to assume the test is running against
those test keys, and will be skipped/ignored when these assertions
fail (which should only occur with `build.snapshot=true`)
Original commit: elastic/x-pack-elasticsearch@871704a3af
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.
Relates elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@abc66ec67d
On Windows a named pipe server must call ConnectNamedPipe() before using
a named pipe. However, if the client has already connected then this
function returns a failure code, with detailed error code
ERROR_PIPE_CONNECTED. The server must check for this, as it means the
connection will work fine. The Java test that emulates what the C++
would do in production did not have this logic.
This was purely a test problem. The C++ code used in production already
does the right thing.
relates elastic/x-pack-elasticsearch#2359
Original commit: elastic/x-pack-elasticsearch@e162887f28
The changes made for elastic/x-pack-elasticsearch#2369 showed that the ML security tests were seriously
weakened by the decision to grant many "minimal" privileges to all users
involved in the tests. A better solution is to override the auth header
such that a superuser runs setup actions and assertions that work by
querying raw documents in ways that an end user wouldn't. Then the ML
endpoints can be called with the privileges provided by the ML roles and
nothing else.
Original commit: elastic/x-pack-elasticsearch@4de42d9e54
Implementation details of ML endpoints should be performed using the
internal client, so that the end user only requires permissions for
the public ML endpoints and does not need to know how they are
implemented. This change fixes some instances where this rule was
not adhered to.
Original commit: elastic/x-pack-elasticsearch@01c8f5172c
* Add support for authz checks at on shard requests
* Add Rest Tests for authorization
* Bulk security - Only reject individual items, rather than a whole shard
* Sync with core change
* Grant "delete" priv in ML smoketest
This role had index and+bulk privileges but it also needs delete (in order to delete ML model-snapshots)
Original commit: elastic/x-pack-elasticsearch@830e89e652
This commit removes some extraneous trailing newlines from
keystore.bash, the packaging test cases for the interaction between
installing X-Pack and the keystore.
Original commit: elastic/x-pack-elasticsearch@86250ecfbc
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.
* TokenService default enabled if HTTP_ENABLED == false
* Fixed tests that need TokenService explicitly enabled
* [DOC] Default value for `xpack.security.authc.token.enabled`
Original commit: elastic/x-pack-elasticsearch@bd154d16eb
As there are two indices to upgrade for watcher, it makes a lot of sense
to also have two upgrade checks.
There is one upgrader for the watches index, which deletes
old templates, adds the new one before and then does the reindexing.
Same for the triggered watches index.
This also means, that there will be two entries popping up in the kibana
UI.
Note: Each upgrade check checks if the other index (for the .watches
upgrade check the triggered watches index and vice versa) is already
upgraded and only if that is true, watcher is restarted.
relates elastic/x-pack-elasticsearch#2238
Original commit: elastic/x-pack-elasticsearch@2c92040ed6
Today we require a pre-shared key to use the token service. Beside the
additional setup step it doesn't allow for key-rotation which is a major downside.
This change adds a TokenService private ClusterState.Custom that is used to distribute
the keys used to encrypt tokens. It also has the infrastructur to add automatic key
rotation which is not in use yet but included here to illustrate how it can work down
the road.
This is considered a prototype and requires additioanl integration testing. Yet, it's fully
BWC with a rolling / full cluster restart from a previous version (also from 5.6 to 6.x)
since if the password is set it will just use it instead of generating a new one.
Once we implement the automatic key rotation via the clusterstate we need to ensure that we are
fully upgraded before we do that.
Also note that the ClusterState.Custom is fully transient and will never be serialized to disk.
Original commit: elastic/x-pack-elasticsearch@1ae22f5d41
Today we require the `bootstrap.password` to be present in the keystore in order to
bootstrap xpack. With the addition of `keystore.seed` we have a randomly generated password
per node to do the bootstrapping. This will improve the initial user experience significantly
since the user doesn't need to create a keystore and add a password, they keystore is created
automatically unless already present and is always created with this random seed.
Relates to elastic/elasticsearch#26253
Original commit: elastic/x-pack-elasticsearch@5a984b4fd8
The rest test waited for the watch to run in the background, but there
were no guarantees that this really happened. Also it waited for five
seconds, instead of just executing the watch manually.
relates elastic/x-pack-elasticsearch#2255
Original commit: elastic/x-pack-elasticsearch@56765a649e
These tests have repeating but not reproducible failures,
where the stash is filled with a second PUT operation and the
watcher stats response does not match. Setting the log to trace
should shed some light on this.
As the smoke tests are only four tests this will not lead to a
log explosion.
Relates elastic/x-pack-elasticsearch#1513, elastic/x-pack-elasticsearch#1874
Original commit: elastic/x-pack-elasticsearch@5832dc7990
Jay Modi
Dimitris Athanasiou
Simon Willnauer
Jason Tedor
Tanguy Leroux
David Roberts
Albert Zaharovits
Yannick Welsch
Boaz Leskes
Tal Levy
Tim Brooks
David Kyle
Tim Vernum
Michael Basnight
Alexander Reelsen