Until now, acking and throttling functionality was applied at the watch level. This has major drawbacks in different aspects:
- When multiple actions are defined on a watch, acking a watch effectively acks all the actions. This is conceptually wrong. Say you have two actions: `email` and `index`. It's very likely you'd like to ack the email action (to avoid receiving too many emails) but at the same time continue indexing the data in the `index` action. Right now it's not possible.
- Different actions types may require different throttling. An `email` action probably needs a longer throttle period compared to an `index` action. Also for different `webhook` actions, the throttling is ultimately determined by the 3rd party system that is called.
This commit changes how we do throttling & acking. Moving this functionality to the action level. Now, when acking, each action in the watch will be acked separately. During executiong, each action will determine whether it needs to be throttled or not. The throttler is not associated with the action, not with the watch.
The throttle period was enhanced. There is a default throttle period that is configured for watcher as a whole (using the `watcher.execution.default_throttle_period` setting. Next to that, each `watch` can define its own `throttle_period` that can serve as the default throttle period for the actions in the watch. Lastly, each action can have its own throttle period set.
Since the throttler is now an action "thing", the `throttle` package was renamed to `throttler` and moved under the `actions` package. Also, `WatchThrottler` was renamed to `ActionThrottler`.
With this change, the `Watch Execute API` changed as well. Now, when executing a watch, you can define an execution mode per action. The execution mode offers 4 types of execution:
- `execute`: executes the watch normally (actually executing the action and it may be throttled)
- `force_execute`: skips/ignores throttling and executes the watch
- `simulate`: simulates the watch execution yet it may be throttled
- `force_simulate`: skips/ignores throttling and simulates the watch execution
As part of this change, the structure of the watch status changed along with the xconent representing the `watch_record`. A new `ActionStatus` was introduced (as part of the `WatchStatus`) and is always set for every action in the watch. This status holds:
- the current state of the action (`ackable`, `awaits_successful_execution`, `acked`)
- the last execution state (success/failure + reason)
- the last successful execution state
- the last throttle state (timestamp + reason)
Original commit: elastic/x-pack-elasticsearch@32c2985ed8
The following additional information will be shown per watch that is executing: `watch_id`, `watch_execution_id`, `triggered_time`, `execution_time`, `execution_phase` (whether it is execution an input, condition or an action) and `stack_trace` (useful for us when a customer reports an issue :) ).
The stats api will by default include the executing watches in the response. In order to control this, a `metric` option has been added, which can be specified as query string argument or as last path element in the stats api url. By default the watcher stats API will only return the basic statistics that are already there.
The `metric` option has the following values:
* `current_watches` - Include the current executing watches in the response.
* `_all` - Include all metrics in the stats response. Not very useful now, but when we expose more metrics in this api it will be more useful.
Original commit: elastic/x-pack-elasticsearch@093bef9bb3
Until today we could not search on the `met` field in the condition result. The reason for that is that this field was index as part of the condition result type only, and we disable the indexing for all condition results (to avoid mapping conflicts).
This commit pulls the `met` condition one level higher and enables its mapping. For now (beta1) we can live with the duplication of the condition result source (were the `met` is not placed in both the condition result type and on the condition result itself). Later we should remove the duplication though.
An example of a "compare" condition result now looks like:
```
"condition": {
"met": true,
"compare": {
"met": true,
"resolved_value": 1
}
}
```
Original commit: elastic/x-pack-elasticsearch@74a3372c25
A simple `condition` that compares a path into the model in the execution context to a value. The comparison is based on the following possible operators: `eq`, `not_eq`, `lt`, `lte`, `gt`, `gte`.
The following example shows a `compare` condition that checks if the total hits in the payload is greater or equal to 5.
```
{
"compare" : {
"ctx.payload.hits.total" : { "gte" : "5" }
}
}
```
Original commit: elastic/x-pack-elasticsearch@6d4f2bbf10
Fixed the mappings of the index action fields such that the `index` and `type` of the index response won't be analyzed.
Original commit: elastic/x-pack-elasticsearch@5db8bf6a33
rely on http header to be send for Shield authentication
and finally omit the manage_watcher role from the test user.
Original commit: elastic/x-pack-elasticsearch@57a6366119
- Disabled the search request body/template fields as they're too volatile to have mappings (applied for both for `search` input and transform)
- Disabled watch level transform result payload
Original commit: elastic/x-pack-elasticsearch@f69b237234
Holds the name of the version as it is defined in the `pom.xml` (different than the version number that is automatically generated by `WatcherVersion`)
Original commit: elastic/x-pack-elasticsearch@b6cf221f63
This change adds a check to make sure that the license is enabled when starting watcher in the tests.
This avoids a race-condition where a test might run before the license manager had a chance to start.
Original commit: elastic/x-pack-elasticsearch@0b9d0da5d4
I think the escaping done in XMustacheFactory (and by extension JsonEscapingMustacheFactory in core) is broken.
You cannot just escape any control character by sticking a '\' in front of it. For example a new line character it '\n' but this will be rendered as a line break. Simply prepending a '\' to this just results in a '\' and then a new line !
Added support for different escaping strategies based on the XContentType of a template for XMustacheEngine.
Currently only JSON escaping is supported using jackson.JsonStringEncoder.
Templates will be prepended with __<contentType>__:: when the content type is set. If this is set to JSON we will json escape the content.
Fixes: elastic/elasticsearch#404
Original commit: elastic/x-pack-elasticsearch@1400cba659
A truststore should not be required as the default system truststore can be used
to validate certificates that have been signed by most commercial CAs.
Additionally, the HttpClient is now a lifecycle component to prevent out of memory
exceptions when starting up with a bad configuration; when an exception is thrown
in the constructor, Guice will continue to try to create the object until the system runs
out of memory.
Closeselastic/elasticsearch#476
Original commit: elastic/x-pack-elasticsearch@2333e47ac1
Transform results may product different payloads per watch/execution. It means that if the resulted transformed payload is mapped and indexed, there's a high chance for mapping conflicts and thus failures.
For this reason, this commit disables the mapping of all `transform` results (on both the watch and the action levels).
This commit also changes the field name of the transform result from `transform_result` to just `transform` (aligned with other result field names - `input`, `condition` and `actions`)
Fixeselastic/elasticsearch#472
Original commit: elastic/x-pack-elasticsearch@2c6d4f5182
Until now you always needed to define the `actions` in a watch. Even if you didn't want any actions, you had to define and empty objects there.
Now, the `actions` is an optional field. When missing, we default to "no actions"
Original commit: elastic/x-pack-elasticsearch@3d1a961232
Move deduping to `validMonthTimes`
This change dedups the `MonthTimes` in `validMonthTimes` to support this two new methods on `MonthTimes` are added `contains(int,DayTime)` which detects if a given day and DayTime combination is contained in the MonthTime and `intersects(MonthTimes)` which detects if a `MonthTimes` has any day/DayTime combinations contained in this `MonthTimes`.
Original commit: elastic/x-pack-elasticsearch@f68039bc39
Code:
- renamed 'executed_request` back to `request` in the search input/transform result (makes more sense in the context of the overall response structure
- renamed `sent_request` back to `request` in the http input result (makes more sense in the context of the overall response structure)
- renamed `http_status` to `status` in the http input result (makes more sense in the context of the overall response structure)
Docs:
- fixed indentations
- fixed wrong example in put-watch docs (`actions` was shown as an array)
- changed action ids to be lowercased & underscored
Original commit: elastic/x-pack-elasticsearch@6ded0936d5
- Renamed `watch_execution` to `execution_result`
- Renamed `actions_results` to `actions`
- Renamed `input_result` to `input`
- Renamed `condition_result` to `condition`
- Updated the `watch_history.json` template to reflect the changes, also added concrete mappings for action types (such that field that should not be analized will be mapped as `not_analyzed`
- Fixed a bug in `WatchUtils.createSearchRequestFromPrototype` where the document types were ignored.
Also, changed the `actions` (fka `actions_results`) from an object to an array. the action id is not part of the action objects (indicated by the `id` field). For example:
```
{
"actions" : [
{
"id" : "my_email",
"email" : {
...
}
}
]
}
```
The reason for this change is to make the path to the action fields predictable deterministic. With the object structure, where the actions were keyed by their `id`, the path to the action fields depended on the action id, which is unpredictable and continuously changing from one action to another. This made it impossible to properly analyze the action data using aggregations (as aggs require full path into the aggregated fields).
With this change, the mappings of `watch_record` changed as well where the `actions` are not defined as nested type, yet it is still configured to include all the fields in the root object. We do this so in the future, when appropriate support will be added to kibana, it'll be able to apply nested aggregations on the actions, enabling correct/safe multi-dimensional aggregations. In the mean time however, while kibana doesn't support nested aggregations, we still need to have all the fields indexed on the root, so at least a single dimensional aggregations can be safely applied.
Also, change the `input` and `condition` objects in the `watch_record` such that their mappings are disabled. The main reason for this is the fact that a lot of the inputs use elements that can be configured in many ways, but the mappings are too strict to accept it. For example, a template can be configured as a `string` or as an `object`.
Original commit: elastic/x-pack-elasticsearch@83464a0c71
This change adds a method to WatcherTestUtils to compare two json structures since order is not guaranteed in template parameters.
The Inline template tests in SearchInputTests and SearchTransformTests now use this instead of string comparision.
Original commit: elastic/x-pack-elasticsearch@c433545128
Also the WatcherUtils now always use the search template support from core and never uses ScriptService:
* A template is re-parsed and extended with the watch context variables.
* A normal request body is converted into a search template and watch context variables are used as template params.
Original commit: elastic/x-pack-elasticsearch@16bacaf094
The following node setting will disable watcher (it's enabled by default).
```
watcher.enabled: false
```
Original commit: elastic/x-pack-elasticsearch@1d0541a924
This change adds a force paramter to delete to allow the forcing of a delete to happen ignoring locking.
This means that watch executions may fail when they go to update the watch. Watches executing from the scheduler will fail fast if the watch they are supposed to execute has been removed.
Also move the history store updates outside of the watch locks.
Fixes: elastic/elasticsearch#405
Original commit: elastic/x-pack-elasticsearch@57561b6f85
moved from `text_body` and `html_body` to a more structured `body` object as follows:
```
{
"body" : {
"text" : "the text body",
"html" : "the html body"
}
}
```
`body` can also accpet a string, in which case it will default to the text body of the email:
```
{
"body" : "the text body of the email"
}
```
the above is a syntactic sugar for the following:
```
{
"body" : {
"text" : "the text body of the email"
}
}
```
Original commit: elastic/x-pack-elasticsearch@92406ac2a1
Bye default, when `attach_data` is set to `true` we attache the watch execution data as a `yaml` file. Now it's possible to configure the format of the file by setting the `attach_data` to an object with a `format` field (can either be set to `yaml` or `json`). Example:
```
"attach_data" : {
"format" : "json"
}
```
Original commit: elastic/x-pack-elasticsearch@5cab59a676
Scripts and Templates are very much alike. There are only two main differences between them when it comes to watcher:
1. The template doesn't have a language type - it's always mustache.
2. Templates are often used to render xcontent (json/yaml). So there's build in support for content type in it
Traditionally, in es core, the configuration of the scripts and templates was always a bit of a mess. There was no consistency between the two and when it comes to the templates configuration, they're even not future proof (future additions to the search requests body can break the templates).
After a long discussion, we've decided that we need to change the way users configure templates & scripts in es core... and align the two. These changes will only come in es 2.0. That said, it's important that watcher will be aligned with es core in how templates and scripts are configured and therefore we need to change it for that purpose.
Watcher will come with support for the scripts & templates configuration format of es 2.0. We want to future proof the configuration from day one.
here are the configuration:
Inline:
```
{
"inline" : "script here",
"lang" : "lang here",
"params" : { parameters here }
}
```
File:
```
{
"file" : "file name here",
"lang" : "language here",
"params" : { parameters here }
}
```
Indexed:
```
{
"id" : "template id here",
"lang" : "language here",
"params" : { parameters here }
}
```
For templates it's the same configuration except there is no `lang` field. Also, for templates, there's native support for xcontent:
```
{
"inline" : { "key" : "{{param1}}" },
"params" : { "param1" : "value1" }
}
}
```
Original commit: elastic/x-pack-elasticsearch@4a31114f35
This change changes how the ScheduledTriggerEvent is parsed to parse using DateMath instead of just as a date. This will allow the execute API to use such constructs as
```
POST _watcher/watch/test_watch/_execute
{
"ignore_throttle" : true,
"trigger_event" : {
"schedule" : {
"triggered_time": "now-5h",
"scheduled_time": "now"
}
}
}
```
Fixes: elastic/elasticsearch#374
Original commit: elastic/x-pack-elasticsearch@fa286b217e
This change changes the manual trigger event to be a wrapper for another `TriggerEvent`.
TriggerEvent is required field for the execute API either via setTriggerEvent on the request or requestBuilder in the java API or `trigger_event` in the REST API
Original commit: elastic/x-pack-elasticsearch@790bdb96a5
This change removes these jars from shading and changes the assemblies file so that they are included as is.
Original commit: elastic/x-pack-elasticsearch@695f5e2b20
Also removed the version checking from the index template support, so that if the settings are changed these settings always get stored in the index template. This means we always put two index templates once watcher started.
And also shutdown watcher after the cluster service has been stopped.
This was done caused deadlock in the integration test framework when we are stopping the test cluster.
If multiple nodes are stopped, a node may briefly try to or run Watcher during the test cluster shutdown.
As part if starting we always put an index template, which will block any other start or stop calls for the WatcherService.
A node was running a put index template call as part of the start procedure while the start framework would stop the node.
The stop call would wait because it doesn't own the lock yet.
The put index template call didn't return, which caused the stop call the keep waiting.
The reason why put index template was hanging is that it failed to return anything while it the node was stopped.
Likely the put index template request got queued up waiting for a change in the cluster state, the cluster service
got stopped and the request got lost, but the watcher start procedure was still waiting.
By stopping Watcher before we stop the cluster service we avoid the scenario described above.
Original commit: elastic/x-pack-elasticsearch@db94b2279e
- wrote an intro where we tell how to get the watcher jar and we tall about the `WatcherClient`
- cleaned up the code snippets
- removed a reference to the `Watch.Parser`
- Also renamed `WatcherStatsResponse#getWatchServiceState` to `watcherStatsResponse.getWatcherState`
Original commit: elastic/x-pack-elasticsearch@814329cba7
An absraction aover XContent byte reference that:
- implements `ToXContent`
- can resolve values given a dot-notation path
- Used in `GetWatchResponse` and `ExecuteWatchResponse`
Also
- Moved `WatchExecutionResult` to the `execution` package
Original commit: elastic/x-pack-elasticsearch@0b41f53f38
- Moved it to its own top level enum `WatcherState`
- Also changed the key holding the current state in the `stats` API to `watcher_state`
Original commit: elastic/x-pack-elasticsearch@2e5ca90f57
Change them to consistently serialize themselves the same way they deserialize themselves.
If the script (or template) is read from a `string` value, it will serialize it self to xcontent as a `string` value as well. Otherwise it will serialize as an object, holding only those fields that where configured in the first place.
Original commit: elastic/x-pack-elasticsearch@52a82e0bbe
This change moves the sanitization of the html body of emails to render time instead of at message send time.
Move the sanitization code the the EmailTemplate class.
Add <hr> tag to allowed html tags.
Add global setting `watcher.actions.email.sanitize_html` that defaults to true. If this is set to false html sanitization will be disabled
Add documentation for sanitization disable setting.
Enhance email tests to verify that sanitization is happening at construction time.
Fixeselastic/elasticsearch#356
Original commit: elastic/x-pack-elasticsearch@282a2d85c2
The body of the http input may hold xcontent data (json/yaml). The problem is that today, the body only accepts text and writing
escapted json in text makes it very hard to maintain.
This commit introduces another field settings - `xbody`. Now, use `xbody` for inlined xcontent and `body` for any text content. When `xbody` is used, we keep around the xcontent type and the http request that is then executed will include a content-type header that will match the xcontent type of input.
Original commit: elastic/x-pack-elasticsearch@7210908064
- added dynamic_templates to the index template that make sure not to analyze email addresses and http `path` and `host` under the `watch_execution` object. This will enabled building aggregations over these fields. Also make sure that different time fields in thw `watch_record` are mapped as date types.
- While at it changed the `watch_execution.input_result` mapping such that it's enabled and only disabled `watch_execution.input_result.payload` as different payloads from different sources may conflict with one another in terms of mappings
- Fixed a bug in the `EmailTemplate` building of the `reply_to` field.
- Also, added missing `execution_time` to the `watch_record` (under `watch_execution` object).
Closeselastic/elasticsearch#335
Original commit: elastic/x-pack-elasticsearch@dd28c70bca
The meta data field was known and `metadata` in the docs but the parsing logic expected `meta`.
This change unifies everything to use `metadata`. Also clean up Watch parsing to fail in a more consistent way when unknown fields are encountered.
Fixeselastic/elasticsearch#329
Original commit: elastic/x-pack-elasticsearch@a9013127d8
This change hijacks the document APIs for the .watches index and prevents direct access to these APIs via REST.
This hijacking is controlled by the "watcher.index.rest.direct_access" setting, this setting defaults to false which restricts access to the index. To allow direct access to the .watches index from the REST API set this setting to `true`
Copied json files from core es to test this feature and added comments to the json files indicating why they have been copied from es core.
Fixeselastic/elasticsearch#336
Original commit: elastic/x-pack-elasticsearch@22335750bd
Also added assertions to verify if all watch record not null field are expected to be there at parse time.
Original commit: elastic/x-pack-elasticsearch@42052ae02c
The problem we have today is that since all write operations on a watch are blocked on the watch lock, if a watch is executing for a long time (for whatever reason) the action will simply block and by that hold a `management` thread for potentially a long time. This impacts elasticsearch as a whole - if for example there's an infinite loop in the condition script, then you try to delete/ack/update the watch 5 times (the default size of the management thread pool in elasticsearch is 5). You effectively block all management threads in elasticsearch and from that moment on, the only thing left to do is kill the node.
This commit introduces a timeout for the tree write watch actions - `put`, `ack` and `delete`. It is set to 10 seconds by default, but you can customize it both in java and in the rest API using the standard `master_timeout` query parameter.
Updated the relevant docs.
Fixeselastic/elasticsearch#309
Original commit: elastic/x-pack-elasticsearch@03292db37b
This change disallows the SCAN search type in our search requests used by search input and search transform.
Add tests for this and update the current tests to both detect SCAN and not use SCAN in valid tests.
Original commit: elastic/x-pack-elasticsearch@c9d61930c8
This is a step between the old and elastic/elasticsearch#184 this change unifies the logic that generates the search requests from prototype for SearchTransforms and SearchInputs.
This change also adds the `executed_request` field to the `SearchTransform.Result` that was missing.
Add tests for SearchTransform.Result parsing and all different templating options for SearchTransform along with testing handling of search_type in SearchTransform which was being missed previously.
Fixeselastic/elasticsearch#311
Original commit: elastic/x-pack-elasticsearch@26e68576f5
This input_result may contain different types for the same field names. This will cause mapping failures when we try to write the updated watch record.
This change disables this field and adds a test in the `ManualExecutionTests` to test this case.
Fixeselastic/elasticsearch#307
Original commit: elastic/x-pack-elasticsearch@5340761343
uboness
Martijn van Groningen
Brian Murphy
jaymode
Shay Banon