The system user needs to be able to execute a reroute action to support the
DiskThresholdDecider. The DiskThresholdDecider executes an empty reroute
request when a high disk watermark has been exceeded.
Relates to elastic/elasticsearch#816
Original commit: elastic/x-pack-elasticsearch@890b44d79e
This change changes the watch specific actions put/get/delete/execute/ack to be watcher.<action>_watch.
Original commit: elastic/x-pack-elasticsearch@86a91cb141
ElasticsearchIntegrationTest now has a method called httpClient with a different return type than the watcher AbstractWatcherTests httpclient.
This change renamed the `AbstractWatcherTests.httpClient` to `AbstractWatcherTests.watcherHttpClient`.
Original commit: elastic/x-pack-elasticsearch@d35f076cf2
A watch may contain sensitive data that typically you don't want to expose in plain text. Exposing means:
- storing it as plain text in the `.watches` and `.watch_history` indices
- storing it in memory in plain text (can be access via mem dump)
- returning it to the user via API in plain text
Examples of such sensitive data:
- The `password` for the email service (can be configured on the watch itself)
- The `password` for http input when using basic auth
- The `passowrd` for webhook action when using basic auth
A new `SecretService` (you heard it right... secret service) was added to handel the secrets across the board. When a watch is first added to watcher, this service converts all the sensitive data to secrets. From that moment on, all sensitive data associated with the watch (whether in stored in the index or in memory) is hidden behind the secret. This service is also used to "reveal" the original sensitive data on-demand when needed (for example, when the email is sent, it is sent with the original text).
There are two implementations for the `SecretService`. The default one is "plain text" where the created secrets don't really hide anything. The second implementation is based on Shield. If Shield is installed and enabled, the `ShieldSecretService` is used which uses shield's crypto service to potentially encrypt the sensitive data (only potentially because Shield's system key must be defined for encryption to take effect, without the system key, the crypto service will not encrypt and instead return the sensitive data in plain text)
Note, even when Shield is installed, the encryption of sensitive data will only be applied if the `watcher.shield.encrypt_sensitive_data` setting is set to `true`. By default it is set to `false`.
The `get watch` and `execute watch` APIs were updated to filter out sensitive data (using special "hide secrets" parameter).
When shield is integrated, we use shield's settings filter to filter out sensitive settings from the REST nodes info API (when shield is not installed or enabled, we don't do this filtering).
For this change several other refactoring needed to take place
- The http auth codebase was refactored to be more modular. Just like with other modular constructs in watcher, we separated `HttpAuth` from `ApplicableHttpAuth` where the former is the configuration construct and tha latter is the applicable ("executable") construct.
- Changed `WatchStore#put` to accept a watch (instead of the watch source). That's more natural way of looking at a store. Also, a `Watch` can now create and return itself as `ByteReference`. In addition, we now don't directly store the watch source as it was sent by the user, instead, we first parse it to a watch (important step to both validate the source and convert all sensitive data to secrets) and then serialize the watch back to `ByteReference`. This way we're sure that only the secrets are stored and not the original sensitive data.
- All `ToXContent` implementation were updated to properly propagate the `Params`
Docs were added to the Shield Integration chapter
Original commit: elastic/x-pack-elasticsearch@4490fb0ab8
SearchInput using inline, indexed and on disk templates.
ScriptedCondition where the script accesses the ctx.
ScriptedCondition where the script throws an exception.
ScriptedCondition where the script doesn't return a boolean.
Webhook tests using templated body, path and parameters.
Some REST test fixes.
Original commit: elastic/x-pack-elasticsearch@d02b6d1d7b
The testChangingAByte method was requesting a random length byte array ranging
from 0-1000. The issue is that a byte array with length of 0 cannot be changed and
therefore this test is not valid in this case. It now requests a range of 1-1000.
Original commit: elastic/x-pack-elasticsearch@88edf535f4
The HttpInputIntegrationTest sometimes unsetted the 'shieldEnabled' field for subsequent test methods, this caused http 401 errors, because the nodes were started with Shield enabled.
Original commit: elastic/x-pack-elasticsearch@b39ca9647d
This change employs the same kind of logic that `HttpClientTest` uses to find an open port for it's server. This should prevent the rare build failures we have seen.
Original commit: elastic/x-pack-elasticsearch@f3b68adad5
If there were many watches concurrently executing that take some time to execute it can a long time to shutdown a node. This change introduces a Timeout when stopping the WatchLockService.
Make watcher lock service timeout configurable. The configuration setting is `watcher.stop.timeout` the default is 30s.
Fixeselastic/elasticsearch#216
Original commit: elastic/x-pack-elasticsearch@06bf029a54
* The validation happens separately from the actual starting and loading. Also the validation happens on the cluster update thread, so that we don't miss any cluster state update.
* The starting/loading part happens on a forked thread, but if it fails then it always retry after a small timeout.
* Simplified the starting code, so that we don't need the callback interface.
Closeselastic/elasticsearch#212
Original commit: elastic/x-pack-elasticsearch@b5cd48e5bb
Before this change DateTimes were being constructed without providing a time zone,
this was causing some non-utc time leakage. In particular watch record ids were being created with non utc dates and
watch records were going to the wrong .watch_history index.
Add Clock.now(DateTimeZone) to allow callers to get UTC now.
Also construct the DateTimes with UTC timezones when we construct from millis.
Add all constuctors of DateTime that do not specify a time zone to the forbidden APIs.
This change makes constructing a `DateTime` object without providing a `DateTimeZone` forbidden.
This is much safer and makes time zone errors much more unlikely to occur.
Statically import DateTimeZone.UTC everywhere it was being used
Now all calls that use DateTimeZone.UTC just refrence UTC.
Fixeselastic/elasticsearch#150
Original commit: elastic/x-pack-elasticsearch@7f23ce605e
Mainly how templates read/write themselves from/to xcontent. Instead of using `text`, use `template`
Original commit: elastic/x-pack-elasticsearch@0d6f317539
* Split the transform into two constructs: `Transform` and `ExecutableTransform`. The former holds all the transform configuration, the latter can execute the transform based on that configuration (an executable transform holds a transform)
- This makes the code clearer to understand and maintain.
- This also enabled to pull some common implementation code into the `ExecutableTransform` and by that reduce the implementation details of each executable to the minimum required.
* Also, extracted the `Transform.Parser` to its own top level class, and renamed it to - `TransformFactory`. The main thing that the factory does is: 1) delegate to the parsing to the `Transform` class, 2) construct & wire up the `ExecutableTransform`.
* With the introduction of `Transform`, we no longer need the `SourceBuilder` for transforms. Instead, we have `Transform.Builder` that help you build a transform. This is much more intuitive from the client perspective.
Original commit: elastic/x-pack-elasticsearch@f6ee0d0c75
* Tune the index templates for higher index throughput.
* Trigger events should be processed and indexed in an async manner.
* Moved the job name into the TriggerEvent
* Added support for fire multiple events at the same time. If multiple events are fired at the same time then use async bulk to persist the watch record.
* Cut simple ticker and timer ticker over to fire multiple events at the same time
* Don't fire more than 1000 events at the time. (this also may result in large bulk requests)
Original commit: elastic/x-pack-elasticsearch@c7f6bd3812
* Changed the isolated ScheduleEngineBenchmark to tests all engines and collect statistics during a test run.
* Updated the isolated schedule benchmark to just test the trigger part of the scheduler and capture some basic matrics.
Original commit: elastic/x-pack-elasticsearch@88bbf3688d
- `TimerScheduleTriggerEngine` - a single threaded Java `Timer`based scheduler. "Ticks" every second and checks all the registered schedules.
- `SimpleTickerScheduleTriggerEngine` - a single threaded scheduler. "Ticks" every second and checks all the registered schedules
- `SchedulerScheduleTriggerEngine` - a single threaded engine based on Java's schedule executor service. Here, every job is added as a scheduled task to the executor and each job is managing its own execution times.
- `HashWheelScheduleTriggerEngine` - a single threaded engine based on Netty's `HashWheelTimer`. Like with the `scheduler` above, every job is added as a scheduled task to the executor and each job is managing its own execution times.
Also:
- Added an undocumented feature to configure the schedule engine in the settings using `watcher.trigger.schedule.engine` (optional values right now are `quartz`, `simple`, `timer`, `hashwheel` and `scheduler`)
- `Cron` is a fork/copy of quartz `CronExpression`.. a bit cleaned up though.
- `Schedule` now exposes `nextScheduledTimeAfter` to return the next scheduled time after the given one.
- `CronnableSchedule` is now based on `Cron` (this exposed bugs in the schedule tests where we generated invalid cron expression. Now, since `Cronnable` creates the actual cron, validation is in place to make sure only valid expressions are created)
- While at it... refactored how the thread pool settings are set. Removed it from the plugin class, now each module is responsible for the settings of its own TPs. Also, if the thread pools are already configured in node settings we don't configure our default ones. This will enable users to configure the TPs in `elasticsearch.yml`
- Also updated `CronEvalTool` to work with `DateTime` construct (instead of java's `Date`)
Original commit: elastic/x-pack-elasticsearch@40d107c66e
Martijn van Groningen
jaymode
Brian Murphy
uboness