Upgrade to Elasticsearch 2.0.0-SNAPSHOT
This moves the master branch to follow Elasticsearch 2.0.0-SNAPSHOT and fixes most problems that occurred during the upgrade. The remaining issues not yet fixed are:
* `HttpClient` and the `Account` used for Email support need to install security manager which is not supported by the elasticsearch security policy. This is not yet resolved an requires fundamental changes and/or a rule in the core policy file. See elastic/elasticsearch#597
* Due to changes to the way Time/Byte settings are parsed settings without a unit must be upgraded. See elastic/elasticsearch#598
* REST tests are currently disabled due to some limitations from Elasticsearch core that don't allow to run 3rd party REST tests. See elastic/elasticsearch#599
Watcher now also inherits the elaticsearch-parent pom file and all it's properties.
Original commit: elastic/x-pack-elasticsearch@1e03234e3e
The execution service will throw an exection if we acquire a watch lock or add a watch to the current executions if Watcher has stopped or is stopping. We should deal with this sitation by properly catching those failures. FYI if a watch managed to put itself to the current executions a shutdown will wait until a watch execution is finished.
Original commit: elastic/x-pack-elasticsearch@5692316072
This change allows the specification of a watch inline to the `_execute` API.
This watch id will not be persisted to the index and if record_execution is set to true it will result in an error.
The internal id `_anonymous_` will be used for the watch id and will be the watch id in the watch record.
Original commit: elastic/x-pack-elasticsearch@00e32c3838
Until now, if the transform failed (either on the watch or action level), an exception would be thrown and it would be captured globally on the watch execution and in the watch record message
This commit bring the error to the transform result
- A new `status` field was added to the transform result. Can either have `success` or `failure` values. When set to `failure` a `reason` field will hold the failure message.
- The `ExecutionService` changed to enable this functionality. Mainly, instead of relying on exception, during the execution the transform result status is checked and if failed the watch execution is aborted.
Original commit: elastic/x-pack-elasticsearch@65b7f51f00
Until now the email sanitization was fixed and could not be configured. That means that if ppl wanted a feature and our sanitization didn't support it, they were forced to disable sanitizaion all together.
In this commit a new `HtmlSanitizer` construct was introduced that is bound by Guice and can be configured via the settings.
Closeselastic/elasticsearch#586
Original commit: elastic/x-pack-elasticsearch@0081d1bf41
Until today connection and read timeout for http was not directly supported. This means that without setting oracle specific system properties at startup, calling a bad http service would by default hold the watch executing thread forever... is niet goed!!!
This commit introduces connection & read timeouts.
- Connection timeouts are timeouts for setting up the connection
- Read timeouts are timeouts waiting for data to be read
By default both timeouts are set to 10 seconds (overriding the default jdk to indefinite). It is possible to customize the default timeouts by settings `watcher.http.default_connection_timeout` and `watcher.http.default_read_timeout` settings).
It is also possible to override these defaults per http request, meaning, per webhook and http input configuration in the watch.
Original commit: elastic/x-pack-elasticsearch@224f50bc8b
Until now if a condition failed to execute (for whatever reason), an exception would be thrown and the watch would just abort. The problem with that approach is while the error message would have been logged in the watch record message, the result of the condition would have been lost.
This commit moves the condition execution error to the condition result (just like we have it today with actions and inputs).
- A `status` field was added to the condition result (can either be `success` or `failure`)
- A `reason` field is added to the condition result in case its status is `failure`
- If the condition result status is `failure`, the watch execution is aborted
Updated the rest APIs to verify the status & type of both the `input` and `condition` results on execution.
Original commit: elastic/x-pack-elasticsearch@dddca03ff5
Until now, if the input failed, an exception would be thrown and it would be captured globally on the watch execution and in the watch recod message. The problem with this approach is that the information about the input is lost. In this commit, the failure is returned as part of the input result.
- A new `status` field was added to the input result. Can either have `success` or `failure` values. When set to `failure` a `reason` field will be set with the error message.
- The `ExecutionService` changed to enable this functionality. Mainly, instead of relying on exception, during the execution the input result is checked for its status and the execution is aborted on failure. Also, the two places where the watch execution is handled were consolidated to a single method `execute(WatchExecutionContext)`.
- Also, the watch execution context id (which will end up being the `watch_record` id) was added the the context model (accessible via scripts and templates). This is done mainly for debugging purposes.
Original commit: elastic/x-pack-elasticsearch@e2567deada
There may be current executions still going on during stopping.
We should try to wait for the current executions to have completed.
Otherwise we can run into a situation where we didn't delete the watch from the .triggered_watches index,
but did insert into the history index. Upon start this can lead to DocumentAlreadyExistsException,
because we already stored the history record during shutdown...
Introduced `CurrentExecutions` to handle this synchronization.
(we always first store the watch record and then remove the triggered watch)
Original commit: elastic/x-pack-elasticsearch@89c4a8d8ad
This commit adds support for indexing multiple documents with the `index` action. This is done by introducing a special `_doc` field. During action execution, the `_doc` field will be looked up in the payload. If found, the value of the field will be considered as the document that needs to be indexed. If the value is an array of objects, each object in that array will be treated as a separate document and all the documents in the array will be bulk indexed.
This commit also changes the result of the action to hold `XContentSource` rather than a payload (to avoid Map creation explosions). Th `XContentSource` was also extended to support lists.
Original commit: elastic/x-pack-elasticsearch@86f454b029
The `compare` condition result will now hold all resolved values, keyed by their associated parameter place holder strings.
Original commit: elastic/x-pack-elasticsearch@f930c77d54
This change adds tests to ack a subset of a watch's actions, use a different throttle period per action in a watch, also adds tests to make sure that both the watch level and global level throttle_period are applied correctly.
Also updates the REST tests to make sure that throttle periods can be set at a watch and action level and are returned from the GET API.
Original commit: elastic/x-pack-elasticsearch@4b006c7830
Until today it was required to provide a trigger event as part of the execute API. There are two issues with it:
1. It's not user friendly (you'd expect that the execute API would just work with just pointing to the watch)
2. The API could expose inconsistencies where on one hand it points to a watch (with a well defined trigger) on the other it enabled the user to provide completely different trigger (of a different type)
This change enable supporting default triggers by enabling the trigger engine to create a simulated trigger event. This enables the execute API to look up the trigger type of the pointed watch, and ask the trigger service to simulate an event for it. It is still possible to override the trigger event data by providing it via the `trigger_data` parameter in the API.
This simplifies the execute API and prepares it for future trigger types as well.
- This commit add missing integration tests for the execute API
- Also, removed unused `setIgnoreThrottling` from the execute request/builder.
Original commit: elastic/x-pack-elasticsearch@b494ae62e6
This change allows the httpinput to receive non json formatted data from a http endpoint
(such as the elasticsearch _cat apis). If non json is read it will be stored in the `payload._value` in the same
way that the `ScriptTransform` handles non map/json data returned by transforming scripts.
Added response_content_type to http input so that the expected content type in the response can be configured. This accepts `yaml`, `json` and `text` but will be overridden by the http headers.
Original commit: elastic/x-pack-elasticsearch@753d37f14e
Now that the watch record is write once and never read/parsed. We can de-normalize its format (the structure of the `watch_record` document) such it'd be optimal for searching and aggregating.
- `execution_result` renamed to `result` (for minimal verbosity)
- the structure of `trigger_event` changed to:
```
{
"trigger_event" : {
"type" : "<trigger_type>",
"triggered_time" : "<datetime>",
"<trigger_type>" : { // type specific data (optional) }
}
}
```
- the structure of `input` results changed to:
```
{
"result" : {
"input" : {
"type" : "<input_type>",
"payload" : { // the payload },
"<input_type>" : { // custom result fields per type (optional) }
},
...
}
}
```
- the structure of `condition` results changed to:
```
{
"result" : {
...
"condition" : {
"type" : "<condition_type>",
"met" : true | false,
"<condition_type>" : { // custom result fields per type (optional) }
},
...
}
}
```
- the structure of `transform` results changed to:
```
{
"result" : {
...
"transform" : {
"type" : "<transform_type>",
"payload" : { // the transformed payload }
"<transform_type>" : { // custom result fields per type (optional) }
},
...
}
}
```
- the structure of `actions` results changed to:
```
{
"result" : {
...
"actions" : [
{
"id" : "<action_id>"
"type" : "<action_type>",
"status" : "success" | "failure" | "simulated" | "throttled",
"reason" : "holds the reasoning if status is either success or throttled",
"transform" : { // action level transform result (if applicable)}
"<action_type>" : { // custom result fields per type (optional) }
},
...
]
}
}
```
Original commit: elastic/x-pack-elasticsearch@98466d4b83
The queued watches metric gives insight into the watches that are queued for execution.
Per watch that is queued, executing information is shared, like the `watch_id`,
when the watch was triggered and when execution started.
Original commit: elastic/x-pack-elasticsearch@deb5ddfde2
Before WatchRecord was used to keep track of an execution of a Watch and used to store actual end results to it before sealing it. (for example build dashboard on the history indices)
The keeping track of an execution has been moved from WatchRecord to TriggeredWatch. If a watch triggers a TriggeredWatch is stored. The TriggeredWatch has its own index and only the watch_id (is part of id), trigger event and state is stored. If the execution of a Watch has finished (regardless if it was successfully) a WatchRecord is stored in a history index and the TriggeredWatch is deleted.
When a watch is getting executedtThe triggered watch is used the create the watch context.
Also the WatchRecord.State has been removed to its own enum class named ExecutionState. The CHECKING value has been removed, because it wasn't really used. The CHECKING state was set when the execution began, but it was never persisted and because of this state has also been removed from triggered watch.
By separating the result of a watch execution we are more flexible to in the future change the document format of WatchRecord. The history indices will be used by users to build analytics on top of watcher. Also the history indices become truely append only indices.
When update the watch status, only change the status part with the update api
Also set the version when we delete the watch on the in memory instance enforce more ensureStarted() in the components
Removed all watch record and result parsing code (actions, conditions, inputs and transforms)
Original commit: elastic/x-pack-elasticsearch@8f5ffdac13
Simon Willnauer
Simon Willnauer
Robert Muir
Martijn van Groningen
uboness
Brian Murphy