This commit changes the IndexLifecycleManager's handling of variables about an index to only update
all of the values at a single time. Previously, all of the index state variables were volatile
members of the IndexLifecycleManager, which meant we could get an inconsistent view of the index
state. Although rare, this is still incorrect so this change adds a single volatile variable that
holds the state as of the last processed cluster state update.
Additionally, the IndexLifecycleManagerIntegTests were updated to have more concurrency and further
stress this portion of the code and its checks.
relates elastic/x-pack-elasticsearch#2973
Original commit: elastic/x-pack-elasticsearch@5f1552b298
This commit adds the manage_index_templates permission to the kibana_system role that is used by
the kibana system user. This is needed due to an upcoming feature in kibana where a index template
will be used to create the saved objects index.
relates elastic/x-pack-elasticsearch#2937
Original commit: elastic/x-pack-elasticsearch@85a67c73aa
Some of our REST tests open many jobs, and assuming each will use 1GB of
RAM on a single node cluster could fail the test. The solution is to
explicitly say the test jobs will use very little RAM.
Original commit: elastic/x-pack-elasticsearch@a3fcfc4589
Currently, any errors that occur during the DeleteExpiredDataAction are logged and the deletion proceeds to the next job. The user will get no indication in the response that something went wrong although nothing should really go wrong unless the cluster is messed up.
This commit changes this so that errors are reported back to the action.
Original commit: elastic/x-pack-elasticsearch@489cf03c3e
This change modifies the way ML jobs are assigned to nodes to primarily
base the decision on the estimated memory footprint of the jobs. The
memory footprint comes from the model size stats if the job has been
running long enough, otherwise from the model memory limit. In addition,
an allowance for the program code and stack is added.
If insufficient information is available to base the allocation decision on
memory requirements then the decision falls back to using simple job
counts per node.
relates elastic/x-pack-elasticsearch#546
Original commit: elastic/x-pack-elasticsearch@b276aedf2f
tool. However, this is a forbidden API so this commit replaces it with URI#getPath. Additionally,
the tests fail with a security manager permission error due to the use of Mockito for exception
throwing. This commit still uses Mockito for throwing exceptions but does it differently in a way
that is acceptable by our test security policy.
Original commit: elastic/x-pack-elasticsearch@5e1d45acf8
Fixes bug when the url option had trailing slashes. The URL built
was invalid (consecutive fwd slashes) but the failure errors of
the subsequent requests were ignored.
URL is built correctly from the option spec.
True HTTP errors and Exceptions are logged and the cmd fails.
relates elastic/x-pack-elasticsearch#2778
Original commit: elastic/x-pack-elasticsearch@62b2d94ca0
Changes to further prepare for feature split with regards to watcher:
- CryptoService has been moved into watcher
- CryptoService.generateKey() has been moved into SystemKeyTools, only
used there
- The creation of the http client/notification classes have been moved
into watcher, no further dependencies on watcher in XPackPlugin
- Each subproject now registers it's own named writeables
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@a60c98ba7e
This problem was introduced due to distributed watch execution.
When a node newer than the master node joins the cluster and gets a
.watches shard assigned it is supposed to start watcher. However
when a new version of the watch history template is part of that new
node (and we might increase that version anytime), this template does
not get installed, because only the master node is updating watcher
templates.
This commit checks if the local node version is higher than the master
node version and then also puts missing templates.
Currently this is done for all watcher templates, not only the watcher history.
relates elastic/x-pack-elasticsearch#2944
Original commit: elastic/x-pack-elasticsearch@4960231ea7
* Add "client-api-objects" dependency for xpack plugin and transport-client
This adds another gradle project, "client-api-objects" which is intended to be a
common dependency so that the xpack plugin and transport-client can share the
same Request and Response objects.
Relates to elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@a6d83299d0
This is related to elastic/elasticsearch#elastic/x-pack-elasticsearch#27388. It modifies x-pack to
be compatible with the removal of the delegating transport channel.
Original commit: elastic/x-pack-elasticsearch@3bd7bf6773
When there were no accounts configured, watcher returned a cryptic
error message containing 'null' in the description. This fix returns
a more clear error message.
On top a dedicated NotificationServiceTests class was added to remove
redundant test cases in the hipchat/jira/slack unit tests, that all
basically tested NotificationService capabilties.
relates elastic/x-pack-elasticsearch#2666
Original commit: elastic/x-pack-elasticsearch@45d0d1df31
* Rename REST spec xpack.deprecation.info to xpack.migration.deprecations
* Fixed parameter-type naming in xpack.ml.get_model_snapshots
* Fixed QS multi-cluster search test to use cluster.remote_info
Original commit: elastic/x-pack-elasticsearch@ccd35b4a6c
Today persistent tasks is only usable from machine learning, but others like ccr will need to use it too.
With this change ccr and other will be able to make use of it too.
Original commit: elastic/x-pack-elasticsearch@c90f01d5f6
This commit adds checks to the TribeWithSecurityIT tests to ensure that the security index is
writeable before making modification operations. Otherwise, we hit errors in tests that are not
always reproducible.
relates elastic/x-pack-elasticsearch#2977
Original commit: elastic/x-pack-elasticsearch@c29bdff7ae
In elastic/x-pack-elasticsearch#2901, the dependency on the tribe module was removed but a few leftover references were missed
in the build.gradle file of the x-pack-elasticsearch plugin. This commit removes these leftover
references.
Original commit: elastic/x-pack-elasticsearch@03f1cae1f5
In order to prepare for separate source directories, this commit moves
a few packages back into the watcher namespaces. A few of them have been
moved out previously as we thought that it might make sense to have a
dedicated notification API. This wont be the case for watcher on ES
anymore, so we can safely move those back into the watcher space.
Packages affected by this move:
* org.elasticsearch.xpack.common.http
* org.elasticsearch.xpack.common.text
* org.elasticsearch.xpack.common.secret
* org.elasticsearch.xpack.common.stats
* org.elasticsearch.xpack.support
* org.elasticsearch.xpack.notification
Tests have been moved accordingly.
The class `XContentUtils` has been split into one implementation for
watcher and one for security as different methods were used.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@0aec64a7e2
Our tests currently rely on waiting for the security index to be available in some cases and in
CI, these checks have been timing out. This commit increases the amount of time that we will wait
for the index before failing to account for slow machines.
Original commit: elastic/x-pack-elasticsearch@639dccd5cb
This checks if `apm-*` indices exist in the cluster to try to determine if APM is in use on the Elasticsearch cluster.
Original commit: elastic/x-pack-elasticsearch@7f9a9a4eee
On Windows, log4cxx always writes to stderr in UTF-16, and we get the
logs from C++ to Java by redirecting stderr to our named pipe. Hence
the log handler in Java needs to tolerate the log stream it's reading
being either UTF-16 (for Windows) or UTF-8 (for other platforms).
Fixeselastic/machine-learning-cpp#385
Original commit: elastic/x-pack-elasticsearch@89237d7125
This kind of sucks, because we shouldn't have to wait that long for tests to run.
But they're failing CI with some regularity, and we rely on these integration tests.
Original commit: elastic/x-pack-elasticsearch@3f4acb2a32
This is a forward-port of elastic/x-pack-elasticsearch/pull/2921.
original commit message:
Before this commit, a cluster with security enabled and backed by
native-realm user permissions allowed rolled upgrades to clusters without
upgrading the `.security` index. This resulted in the newly established
6.0 cluster not able to register the native-realm users previously established
in the `.security` index. In order to fix this, one would have to rely on file-based
users to re-configure and upgrade the `.security` index. Since this state is easily
avoidable with an upgrade, this commit rejects the joining of upgraded nodes without
upgrading the security index beforehand.
modifications:
Test with 7.x vs 6.x nodes.
Original commit: elastic/x-pack-elasticsearch@56f81bfb20
This commit updates the logic for determining which branch to use to make it consistent with the
logic in elasticsearch. This change means that testing BWC within the same major picks the correct
branch.
Original commit: elastic/x-pack-elasticsearch@2d75d15c41
This adds a rolling upgrade test for X-Pack monitoring. It works by using the `_xpack/monitoring/_bulk` endpoint to send arbitrary data, then verify that it exists.
This forces a few things to happen, thereby testing the behavior:
1. The templates must exist.
2. The elected master node must be "ready" to work (hence the first
point).
3. The same "system_api_version" is accepted by every version of ES.
Original commit: elastic/x-pack-elasticsearch@012e5738bb
Improving logging for unexpected autodetect termination (crash, oom). Output to the log pipe not conforming to the json log output format are treated as fatal error and logged, so that the crash as well as a proper error message if available gets logged.
Original commit: elastic/x-pack-elasticsearch@ae5d792d3f
This change should have been made in elastic/x-pack-elasticsearch#2913. Now we hold the process
context lock throughout the job close procedure, the timeout for trying
to lock it should be the timeout used for job open/close rather than the
timeout for connecting named pipes.
Original commit: elastic/x-pack-elasticsearch@79672b0825
This change removes the xpack plugin's dependency on the tribe module, which is not a published
artifact. For the most part this just involves moving some test classes around, but for the
security and tribe integration the usage of constant settings was removed and replaced with the
string names. This is a bit unfortunate, but a test was added in a QA project that depends on tribe
that will alert us if a new setting is added that we need to be aware of.
relates elastic/x-pack-elasticsearch#2656
Original commit: elastic/x-pack-elasticsearch@649a8033e4
Jay Modi
David Roberts
Hendrik Muhs
Simon Willnauer
Dimitris Athanasiou
Luca Cavanna
Michael Basnight
Nhat Nguyen
jaymode
Albert Zaharovits
Alexander Reelsen
Tim Brooks
Lee Hinman
Clinton Gormley
Martijn van Groningen
Chris Earle
Ryan Ernst
Tim Vernum
Tal Levy