When a watch is acknowledged, while it is also being executed, the
acknowledgment information can get lost. The reason for this is the
fact, that the execution writes the watch status inside of the watch
regardless, if other writes happened inbetween to make sure the
execution state is caught.
This commit checks the current executions in the execution service and
aborts the API call, if the specified watch ID can be found in those.
Note, this does not prevent this issue fully, as a watch could be
triggered, while the acknowledgement update is running, but it does
reduce the surface area of this problem. In order to properly solve
this, indexing the watch status as part of a watch would need to be
changed.
relates elastic/x-pack-elasticsearch#4003
Original commit: elastic/x-pack-elasticsearch@d7e218b2ac
Many users take the JSON from an PUT watch API and put it into the
execute watch API and then start to wonder why there is an error, as
they forget to wrap the watch inside a 'watch' field.
This commit adds a better error message in this case hinting at the user
to add a 'watch' field.
Original commit: elastic/x-pack-elasticsearch@5b56b4abad
The credentials now get injected via environment variables, so that
external services can pull those.
As soon as the specified environment variables are set, the tests are run. No need to check for the @Network annotation
This also introduces new secret store settings for the secure settings in order to be sure to not leak them in the configuration files, that get dumped.
Relates elastic/x-pack-elasticsearch#3800
Original commit: elastic/x-pack-elasticsearch@a2cfb9cb86
The HTTPClient in watcher always appended a question mark at the end of
an URL, regardless if parameters were used or not. This commit adds a
check to only pass valid parameters to the URI construction.
Original commit: elastic/x-pack-elasticsearch@184f8f441c
If a user specifies an 'Authorization' header in an HTTPRequest we,
which might be needed instead of using HTTP Basic Auth due to using
Bearer Authentication, then in case of an failure, the request gets
logged including that Authorization header.
In addition, each implementation of a sent message for jira/hipchat/slack filters
out special fields when a HTTP request is written in case of a failed
response in order to not leak secret data.
Relates elastic/x-pack-elasticsearch#3800
Original commit: elastic/x-pack-elasticsearch@66efdd9b36
This change disables security for trial licenses unless security is
explicitly enabled in the settings. This is done to facilitate users
getting started and not having to deal with some of the complexities
involved in getting security configured. In order to do this and avoid
disabling security for existing users that have gold or platinum
licenses, we have to disable security after cluster formation so that
the license can be retrieved.
relates elastic/x-pack-elasticsearch#4078
Original commit: elastic/x-pack-elasticsearch@96bdb889fc
Aggregations may return `NaN`, and the comparison code would return `true` if this result was passed to a `gte` or `lte` condition.
Original commit: elastic/x-pack-elasticsearch@3b16ae6675
If a watch is not active, it should still be executed, if it is called
via the execute watch API.
This commit adds an additional method to the execution context to check
for this, which returns true for a manual execution context but checks
the watch status for the triggered one.
Original commit: elastic/x-pack-elasticsearch@18f3f9e84b
This ensures that the per watch stats in the trigger service are
properly cleared, when execution is paused.
Original commit: elastic/x-pack-elasticsearch@f5119e4072
This adds back usage stats by pickybacking on the watcher stats, which
are already running distributed in order to collect and merge watcher
statistics.
In order to be able to track statistics, we need to add information for
each watch in an in-memory data structure that is processed whenever a
usage request is coming in. This processing creates a number of counters
for each node, which then are merged together in the usage stats.
relates elastic/x-pack-elasticsearch#4071
Original commit: elastic/x-pack-elasticsearch@c8bfed288f
The toXContent serialization of the transform input was broken, which
could lead to the bad case that a watch could be stored with an invalid
toXContent serialization, that could not be read again, when the watch
should either be executed or even just returned by the Get watch API.
relates elastic/x-pack-elasticsearch#4049
Original commit: elastic/x-pack-elasticsearch@f06ad768b8
* Decouple XContentBuilder from BytesReference
This commit handles the removal of all mentions of BytesReference from
XContentBuilder. This is needed so that we can completely decouple the XContent
code and move it into its own dependency.
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28972
Original commit: elastic/x-pack-elasticsearch@8ba2e97b26
This wraps the stream (`.streamInput()`) that is passed to many of the
`createParser` instances in the enclosing (or a new) try-with-resources block.
This ensures the `BytesReference.streamInput()` is closed.
Relates to elastic/x-pack-elasticsearch#28504
Original commit: elastic/x-pack-elasticsearch@7546e3b4d4
* Pass InputStream when creating XContent parser
Rather than passing the raw `BytesReference` in when creating the xcontent
parser, this passes the StreamInput (which is an InputStream), this allows us to
decouple XContent from BytesReference.
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28754
* Use the streamInput variant, not sourceAsString
Original commit: elastic/x-pack-elasticsearch@dd5d8b1654
The watcher thread pool is scaled by the number of CPUs and has by
default up to 5x the number of cores. This is needed because we assumme
I/O based waiting workloads for watcher. However if the node is not a
data node, there will not be any execution of watches with the exception
of a user calling the execute watch API on that node.
This means, we can get away with just one thread, so that there is no
need for the JVM to manage more threads on master/client or ingest only
nodes.
Original commit: elastic/x-pack-elasticsearch@b5899401d3
The current toXContent serialization of a failed hipchat message writes
the same field called status twice and thus cannot be stored in the
watch history.
This commit ensures the field gets only written once.
relates elastic/x-pack-elasticsearch#3919
Original commit: elastic/x-pack-elasticsearch@fb499e8055
... yet support updates. This commit introduces a few changes of how
watches are put.
The GET Watch API will never return credentials like basic auth
passwords, but a placeholder instead now. If the watcher is enabled to
encrypt sensitive settings, then the original encrypted value is
returned otherwise a "::es_redacted::" place holder.
There have been several Put Watch API changes.
The API now internally uses the Update API and versioning. This has
several implications. First if no version is supplied, we assume an
initial creation. This will work as before, however if a credential is
marked as redacted we will reject storing the watch, so users do not
accidentally store the wrong watch.
The watch xcontent parser now has an additional methods to tell the
caller if redacted passwords have been found. Based on this information
an error can be thrown.
If the user now wants to store a watch that contains a password marked
as redacted, this password will not be part of the toXContent
representation of the watch and in combinatination with update request
the existing password will be merged in. If the encrypted password is
supplied this one will be stored.
The serialization for GetWatchResponse/PutWatchRequest has changed.
The version checks for this will be put into the 6.x branch.
The Watcher UI now needs specify the version, when it wants to store a
watch. This also prevents last-write-wins scenarios and is the reason
why the put/get watch response now contains the internal version.
relates elastic/x-pack-elasticsearch#3089
Original commit: elastic/x-pack-elasticsearch@bb63be9f79
* [Tests] Ensure that Watcher templates are created before creating alias
The createWatcherIndicesOrAliases() method randomly created watcher
indices with or without aliases. On slow machines it is possible that
the indices get created before WatcherIndexTemplateRegistry put the
templates. When it happens, the indices will be created without the
right index.format value, preventing the WatcherService to start (as
it checks the index.format before starting, in case an upgrade is
required)
relates elastic/x-pack-elasticsearch#3965
Original commit: elastic/x-pack-elasticsearch@82946a2be0
In order to prevent exceptions to bubble up to the thread pool exception
handler, this properly wraps all the calls for the watcher service
within an executor into an AbstractRunnable to catch and log a possible
exception.
relates elastic/x-pack-elasticsearch#3854
Original commit: elastic/x-pack-elasticsearch@c0b39e6b5b
This commit introduces a new watcher setting to defer starting watcher
until it has been called with the API for the first time. This is
primarily useful in testing environments, as this ensures that watcher
does not try to reload itself because of starting first and then
creating watcher indices.
In addition the undocumented and unused option
xpack.watcher.start_immediately has been removed.
Relates elastic/x-pack-elasticsearch#3854
Original commit: elastic/x-pack-elasticsearch@2b55aec4ad
This fixes a regression introduced in Elasticsearch 6.0, when switching
from HttpURLConnection to the Apache HTTP Client.
In the old implementation there was no way to specify if you wanted to use HTTP
or HTTPS for your proxy, only HTTP. If people needed to use HTTPs, they
could just use the CONNECT feature of the proxy.
The new implementation used the scheme of the request that was about to
be sent out as the proxy scheme to be used. So if the request was HTTPS
but the proxy server was HTTP this created a problem.
This commit changes the default scheme to be just HTTP, so that then the
standard CONNECT procecure is taken care off.
Without a real proxy server this is super hard to test. I have verified
this with the following test against a tinyproxy running on port 8888,
but I do not have a great idea how to test this in a unit testable way using a real proxy.
Original commit: elastic/x-pack-elasticsearch@f68e72d8f1
The watcher index uses auto expanding of replicas nowadays, so there is
no need to set the number of replicas.
Also ensuring that all watcher indices are green on startup has been moved
to one call, in order to prevent indices being moved around after adding
the triggered watches or watch history index.
Original commit: elastic/x-pack-elasticsearch@dbe04adf64
java.time features it's own halted clock, called a fixed clock, we can
use that one.
On top of that the watcher xcontent parser does not need a clock at all,
just a timestamp when parsing happened.
Original commit: elastic/x-pack-elasticsearch@2061aeffe1
This adds support to allow different schemes in a proxy being used
compared to what the actual request requires. So if your proxy runs via
HTTP, but the endpoint you want to connect to uses HTTPS, this is now
possible to configure the proxy explicitely.
Also a small unit test for parsing this has been added.
relates elastic/x-pack-elasticsearch#3596
Original commit: elastic/x-pack-elasticsearch@176f7cdf0e
Tests have been failing because of out of order cluster state processing
or because of stopping/starting was still in progress. Current tests do
not do further stop/start tries after the first leading to potential
being stuck in an unwanted state. This commit removes the methods
checking for the state being started or stopped in favor of using the
stopWatcher/startWatcher methods which now check if the desired state is
reached and otherwise issue another start/stop command.
Original commit: elastic/x-pack-elasticsearch@97b3232a6a
The WatcherLifeCycleService is responsible for deciding if watches need
to be reloaded while running. In order to do this, the service stores
the currently local shard allocation ids in a List.
This data structure however was not properly updated all the time, when
it should have been - for example when a master node is not available.
This lead to unintended reloads, even though there was no change in the
allocated shards. This in turn lead to unwanted executions and unwanted
loading of triggered watches.
This commit should also fix one of the more nasty ongoing test failures,
where the test returns with an exception that only parts of watcher have
been started. The AbstractWatcherIntegrationTestCase now properly waits
until watcher is started before starting the test case itself.
Original commit: elastic/x-pack-elasticsearch@097f12a900
In order to more easily integrate xpack once it moves into the
elasticsearch repo, references to the existing x-pack-elasticsearch need
to be reduced. This commit introduces a few helper "methods" available
to any project within xpack (through gradle project extension
properties, as closures). All refeerences to project paths now use these
helper methods, except for those pertaining to bwc, which will be
handled in a followup.
Original commit: elastic/x-pack-elasticsearch@850668744c
The EmailSecretsIntegrationTests did not properly wait for watcher to be
stopped before starting it again.
This test failed a few times per week across master and 6.x branches.
Original commit: elastic/x-pack-elasticsearch@cf20d58b0b
Alexander Reelsen
Lee Hinman
David Kyle
Jay Modi
Lee Hinman
Alan Woodward
Luca Cavanna
Tanguy Leroux
Adrien Grand
Tim Brooks
Martijn van Groningen
Ryan Ernst
Michael Basnight