* Use XPackRestIT as base class for XDocsClientYamlTestSuiteIT
* Remove the XPackRestTestCase class
* Address review comments
* Fix checkstyle checks
Original commit: elastic/x-pack-elasticsearch@c2a5e60c12
In order to support the repository split, this changes the
`AbstractWatcherIntegrationTestCase` to not run with security enabled.
We have a dedicated QA project called `smoke-test-watcher-with-security`,
where tests that explicitely need security should be running.
This commit removes the possibility to enable security as part of the
test case. In addition some tests have been moved over to the dedicated
project.
In addition the `timewarp` functionality cannot be configured with a
system property anymore. This would not have worked anyway, because
tests were already dependent on that functionality and did not have any
other means of running. A bit of redundant code was removed due to this.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@b24b365ad1
The chained input in watcher is a useful feature to
call several endpoints before execution a condition.
However it was pretty hard to modify data from a previous
input in order to be able to execute it in another input.
This commit adds a another input, called a `transform` input,
which allows you to do a transform as another input in a chained
input.
See this example
```
"input" : {
"chain" : {
"inputs" : [ <1>
{
"first" : {
"simple" : { "path" : "/_search" }
}
},
{
"second" : {
"transform" : {
"script" : "return [ 'path' : 'ctx.payload.first.path' + '/' ]"
}
}
},
{
"third" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : 9200,
"path" : "{{ctx.payload.second.path}}" <2>
}
}
}
}
]
}
}
```
This allows for far more flexibility before executing the next input in a chained
one.
Original commit: elastic/x-pack-elasticsearch@3af9ba6e9b
In order to be able to execute a watch as the user, who stored the
watch, this commit stores certain headers of the thread context, that
was used when the watch was stored.
Upon loading the watch the headers are loaded and applied for the
following watcher execution features
* search transform
* search input
* index action
A special case is the execute watch API, which overrides the headers loaded
from the watch with the one of the current request, so that a user
cannot execute this watch with other privileges of the user who stored it.
Only the headers "es-security-runas-user", "_xpack_security_authentication" are
copied for now, as those are needed for our security features.
The headers are stored in watch status in the watch and are not returned by default,
when the GET Watch API is used. A search reveals those of course.
relates elastic/x-pack-elasticsearch#2201
Original commit: elastic/x-pack-elasticsearch@9803bd51c2
Fixes to the build system, particularly around BWC testing, and to make future
version bumps less painful.
Original commit: elastic/x-pack-elasticsearch@a1d456f30a
This change removes the InternalClient and the InternalSecurityClient. These are replaced with
usage of the ThreadContext and a transient value, `action.origin`, to indicate which component the
request came from. The security code has been updated to look for this value and ensure the
request is executed as the proper user. This work comes from elastic/x-pack-elasticsearch#2808 where @s1monw suggested
that we do this.
While working on this, I came across index template registries and rather than updating them to use
the new method, I replaced the ML one with the template upgrade framework so that we could
remove this template registry. The watcher template registry is still needed as the template must be
updated for rolling upgrades to work (see elastic/x-pack-elasticsearch#2950).
Original commit: elastic/x-pack-elasticsearch@7dbf2f263e
Changes to further prepare for feature split with regards to watcher:
- CryptoService has been moved into watcher
- CryptoService.generateKey() has been moved into SystemKeyTools, only
used there
- The creation of the http client/notification classes have been moved
into watcher, no further dependencies on watcher in XPackPlugin
- Each subproject now registers it's own named writeables
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@a60c98ba7e
* Rename REST spec xpack.deprecation.info to xpack.migration.deprecations
* Fixed parameter-type naming in xpack.ml.get_model_snapshots
* Fixed QS multi-cluster search test to use cluster.remote_info
Original commit: elastic/x-pack-elasticsearch@ccd35b4a6c
This commit adds checks to the TribeWithSecurityIT tests to ensure that the security index is
writeable before making modification operations. Otherwise, we hit errors in tests that are not
always reproducible.
relates elastic/x-pack-elasticsearch#2977
Original commit: elastic/x-pack-elasticsearch@c29bdff7ae
In order to prepare for separate source directories, this commit moves
a few packages back into the watcher namespaces. A few of them have been
moved out previously as we thought that it might make sense to have a
dedicated notification API. This wont be the case for watcher on ES
anymore, so we can safely move those back into the watcher space.
Packages affected by this move:
* org.elasticsearch.xpack.common.http
* org.elasticsearch.xpack.common.text
* org.elasticsearch.xpack.common.secret
* org.elasticsearch.xpack.common.stats
* org.elasticsearch.xpack.support
* org.elasticsearch.xpack.notification
Tests have been moved accordingly.
The class `XContentUtils` has been split into one implementation for
watcher and one for security as different methods were used.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@0aec64a7e2
This adds a rolling upgrade test for X-Pack monitoring. It works by using the `_xpack/monitoring/_bulk` endpoint to send arbitrary data, then verify that it exists.
This forces a few things to happen, thereby testing the behavior:
1. The templates must exist.
2. The elected master node must be "ready" to work (hence the first
point).
3. The same "system_api_version" is accepted by every version of ES.
Original commit: elastic/x-pack-elasticsearch@012e5738bb
This change removes the xpack plugin's dependency on the tribe module, which is not a published
artifact. For the most part this just involves moving some test classes around, but for the
security and tribe integration the usage of constant settings was removed and replaced with the
string names. This is a bit unfortunate, but a test was added in a QA project that depends on tribe
that will alert us if a new setting is added that we need to be aware of.
relates elastic/x-pack-elasticsearch#2656
Original commit: elastic/x-pack-elasticsearch@649a8033e4
Halt OpenLDAP fixture after :x-pack-elasticsearch:qa:openldap-tests:test
Currently the OpenLDAP vagrant fixture is not halted.
Reruning the test will fail because the new fixture instance will try to bind to
the same host ports. Project :x-pack-elasticsearch:qa:openldap-tests:test is
the only one using the OpenLDAP fixture from
:x-pack-elasticsearch:test:openldap-fixture.
relates elastic/x-pack-elasticsearch#2619
Original commit: elastic/x-pack-elasticsearch@bea2f81b76
This change fixes the check for the version of the security template after the template updater was
changed to only run on the master node in elastic/elasticsearch#27294. Additionally, the wait time
for the cluster to have a yellow status has been increased to account for delayed shards and slower
machines.
Original commit: elastic/x-pack-elasticsearch@a2e72bed12
This is the X-Pack side of elastic/elasticsearch#27235. To force people
who construct an Environment object in production code to think about the
correct setting of configPath there is no longer a single argument
constructor in the Environment class. Instead there is a factory method
in the test framework to replace it. Having this in the test framework
ensures that there is no way to use it in production code.
Original commit: elastic/x-pack-elasticsearch@4860e92d90
This commit removes the FAILED state for the IndexAuditTrail so that we always try to keep starting
the service. Previously, on any exception during startup we moved to a failed state and never tried
to start again. The users only option was to restart the node. This was problematic in the case of
large clusters as there could be common timeouts of cluster state listeners that would cause the
startup of this service to fail.
Additionally, the logic in the IndexAuditTrail to update the template on the current cluster has
been removed and replaced with the use of the TemplateUpgradeService. However, we still need to
maintain the ability to determine if a template on a remote cluster should be PUT. To avoid always
PUTing the template, the version field has been added so it only needs to be PUT once on upgrade.
Finally, the default queue size has been increased as this is another common issue that users hit
with high traffic clusters.
relates elastic/x-pack-elasticsearch#2658
Original commit: elastic/x-pack-elasticsearch@27e2ce7223
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.
This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.
The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.
Original commit: elastic/x-pack-elasticsearch@981f7a40e5
This commit replaces the REST test that the global checkpoint sync
action runs successfully as a privileged user. The test needs to be
replaced because it has a small race condition. Namely, the check that
the post-operation global checkpoint sync was successful could run
before the sync finishes running. To address this, we replace the REST
test with a test where we have a little more control and can assert busy
to avoid this race from failing the test.
Relates elastic/x-pack-elasticsearch#2749
Original commit: elastic/x-pack-elasticsearch@ea585b843c
After a write operation on an index, a post-operation global checkpoint
sync fires. Previously, this action fired on the same user as executed
the write action. If the user did not have priviledges for this action,
the global checkpoint sync would fail. With an upstream change in core,
this action now fires as the system user. This commit adds a test that
create a user that has minimal write permissions on an index, but none
that would imply it could execute the global checkpoint sync. This then
serves as a test that the upstream change to fire the global checkpoint
sync as the system user is correct. This test must run as a mulit-node
test so that a replica is a assigned so that the global checkpoint sync
fires in the first place. This test does indeed fail without the
upstream change, and passes with it.
Relates elastic/x-pack-elasticsearch#2744
Original commit: elastic/x-pack-elasticsearch@bf7e771756
This commit fixes indentation in certgen.bash, adds a check on cluster
health in bootstrap_password.bash and fixes a bug in xpack.bash
Original commit: elastic/x-pack-elasticsearch@d6847f6640
Adds the GET overall_buckets API.
The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets
The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.
relates elastic/x-pack-elasticsearch#2693
Original commit: elastic/x-pack-elasticsearch@ba6061482d
Do not execute bind on on the LDAP reader thread
Each LDAP connection has a single associated thread, executing the handlers for async requests; this is managed by the LDAP library. The bind operation is blocking for the connection. It is a deadlock to call bind, if on the LDAP reader thread for the same connection, because waiting for the bind response blocks the thread processing responses (for this connection).
This will execute the bind operation (and the subsequent runnable) on a thread pool after checking for the conflict above.
Closes: elastic/x-pack-elasticsearch#2570, elastic/x-pack-elasticsearch#2620
Original commit: elastic/x-pack-elasticsearch@404a3d8737
Since elastic/elasticsearch#26878, array and list of settings are
internally represented as actual lists. This makes filtering works
as expected when it comes to filter out arrays/lists.
The packaging tests used to check the presence of the XPack SSL
certificated_authorities setting which should have always been filtered.
By fixing the filtering of settings, elastic/elasticsearch#26878 broke
this packaging test.
This commit changes this test so that it does not expect certificated_authorities
setting to exist in the Nodes Info response.
relates elastic/x-pack-elasticsearch#2688
Original commit: elastic/x-pack-elasticsearch@cb299186b8
The true purpose of this test is to introduce another test alongside
the original, so that the test suite passes even if the other test
is skipped due to the assumption it makes about `build.snapshot`.
Original commit: elastic/x-pack-elasticsearch@709d7a5dc5
Release tests were introduced that sets the `build.snapshot`
system property to `false` to mimic release builds. This invalidates
the hardcoded license signatures that were signed against the
integration test pub/priv keys. This commit modifies the
license-validation assertions to assume the test is running against
those test keys, and will be skipped/ignored when these assertions
fail (which should only occur with `build.snapshot=true`)
Original commit: elastic/x-pack-elasticsearch@871704a3af
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.
Relates elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@abc66ec67d
On Windows a named pipe server must call ConnectNamedPipe() before using
a named pipe. However, if the client has already connected then this
function returns a failure code, with detailed error code
ERROR_PIPE_CONNECTED. The server must check for this, as it means the
connection will work fine. The Java test that emulates what the C++
would do in production did not have this logic.
This was purely a test problem. The C++ code used in production already
does the right thing.
relates elastic/x-pack-elasticsearch#2359
Original commit: elastic/x-pack-elasticsearch@e162887f28
The changes made for elastic/x-pack-elasticsearch#2369 showed that the ML security tests were seriously
weakened by the decision to grant many "minimal" privileges to all users
involved in the tests. A better solution is to override the auth header
such that a superuser runs setup actions and assertions that work by
querying raw documents in ways that an end user wouldn't. Then the ML
endpoints can be called with the privileges provided by the ML roles and
nothing else.
Original commit: elastic/x-pack-elasticsearch@4de42d9e54
David Kyle
Igor Motov
Alexander Reelsen
David Turner
Jay Modi
Zachary Tong
Clinton Gormley
jaymode
Chris Earle
Albert Zaharovits
David Roberts
Jason Tedor
Dimitris Athanasiou
Simon Willnauer
Tanguy Leroux
Yannick Welsch
Boaz Leskes
Tal Levy
Tim Brooks