Commit Graph

701 Commits

Author SHA1 Message Date
Simon Willnauer 501fd2164d make all rest tests pass
Original commit: elastic/x-pack-elasticsearch@b7a108b077
2015-06-16 17:23:47 +02:00
Simon Willnauer d175a75cfc no sec manager for now
Original commit: elastic/x-pack-elasticsearch@36329494ea
2015-06-16 16:28:47 +02:00
Martijn van Groningen e29df8dd60 Added inline watch support to _execute API
This change allows the specification of a watch inline to the `_execute` API.
This watch id will not be persisted to the index and if record_execution is set to true it will result in an error.
The internal id `_anonymous_` will be used for the watch id and will be the watch id in the watch record.

Original commit: elastic/x-pack-elasticsearch@00e32c3838
2015-06-16 16:10:07 +02:00
Simon Willnauer 4cfdf620e5 make tests pass witout security manager
Original commit: elastic/x-pack-elasticsearch@59931382d5
2015-06-16 15:16:21 +02:00
Simon Willnauer 9019c979f3 fix permissions for REST tests
Original commit: elastic/x-pack-elasticsearch@a157980a47
2015-06-16 14:10:15 +02:00
Simon Willnauer 69b65f613a don't use DNS resolution to test unknown domains
Original commit: elastic/x-pack-elasticsearch@953f840a99
2015-06-16 14:04:43 +02:00
Simon Willnauer 0d757413d5 make other slow=false tests pass
Original commit: elastic/x-pack-elasticsearch@1aee8b7744
2015-06-16 13:53:15 +02:00
Simon Willnauer 491e8fc167 more test fixes - more nocommits
Original commit: elastic/x-pack-elasticsearch@9d230d3b71
2015-06-16 10:08:24 +02:00
Martijn van Groningen d3bd643bf4 Fix HttpClientTests, TriggeredWatchStoreTests and WatchStoreTests.
Original commit: elastic/x-pack-elasticsearch@efd0291a09
2015-06-15 23:08:48 +02:00
Martijn van Groningen 2bb03cba89 Fix rest tests
Original commit: elastic/x-pack-elasticsearch@e92139512b
2015-06-15 23:05:01 +02:00
Martijn van Groningen 72fed1be7f Fix Shield related errors
Original commit: elastic/x-pack-elasticsearch@8a2f1e0586
2015-06-15 23:04:40 +02:00
Simon Willnauer e03b4c1b83 some progress while shield is still fucked up
Original commit: elastic/x-pack-elasticsearch@90402b7ba8
2015-06-15 18:43:16 +02:00
uboness 03b704f79b Moved transform errors to the tranform result
Until now, if the transform failed (either on the watch or action level), an exception would be thrown and it would be captured globally on the watch execution and in the watch record message

This commit bring the error to the transform result

- A new `status` field was added to the transform result. Can either have `success` or `failure` values. When set to `failure` a `reason` field will hold the failure message.
- The `ExecutionService` changed to enable this functionality. Mainly, instead of relying on exception, during the execution the transform result status is checked and if failed the watch execution is aborted.

Original commit: elastic/x-pack-elasticsearch@65b7f51f00
2015-06-15 17:28:52 +02:00
uboness 09fcecc069 Made email html body sanitization configurable
Until now the email sanitization was fixed and could not be configured. That means that if ppl wanted a feature and our sanitization didn't support it, they were forced to disable sanitizaion all together.

In this commit a new `HtmlSanitizer` construct was introduced that is bound by Guice and can be configured via the settings.

Closes elastic/elasticsearch#586

Original commit: elastic/x-pack-elasticsearch@0081d1bf41
2015-06-15 16:44:07 +02:00
Simon Willnauer 78c0159949 make tests pass forbidden APIs
Original commit: elastic/x-pack-elasticsearch@985fc6d2ff
2015-06-15 15:45:35 +02:00
uboness b6e8df6a32 Added connection/read timeout support for HTTP
Until today connection and read timeout for http was not directly supported. This means that without setting oracle specific system properties at startup, calling a bad http service would by default hold the watch executing thread forever... is niet goed!!!

This commit introduces connection & read timeouts.

- Connection timeouts are timeouts for setting up the connection
- Read timeouts are timeouts waiting for data to be read

By default both timeouts are set to 10 seconds (overriding the default jdk to indefinite). It is possible to customize the default timeouts by settings `watcher.http.default_connection_timeout` and `watcher.http.default_read_timeout` settings).

It is also possible to override these defaults per http request, meaning, per webhook and http input configuration in the watch.

Original commit: elastic/x-pack-elasticsearch@224f50bc8b
2015-06-15 15:33:40 +02:00
Simon Willnauer fc95323904 first cut - everything compiles
Original commit: elastic/x-pack-elasticsearch@f29fd6de3f
2015-06-15 14:53:03 +02:00
uboness a9448ab2a4 Moved condition errors to their result
Until now if a condition failed to execute (for whatever reason), an exception would be thrown and the watch would just abort. The problem with that approach is while the error message would have been logged in the watch record message, the result of the condition would have been lost.

This commit moves the condition execution error to the condition result (just like we have it today with actions and inputs).

- A `status` field was added to the condition result (can either be `success` or `failure`)
- A `reason` field is added to the condition result in case its status is `failure`
- If the condition result status is `failure`, the watch execution is aborted

Updated the rest APIs to verify the status & type of both the `input` and `condition` results on execution.

Original commit: elastic/x-pack-elasticsearch@dddca03ff5
2015-06-13 04:22:31 +02:00
uboness 0ac5dc8271 [lib] - upgraded java mail to 1.5.3
Original commit: elastic/x-pack-elasticsearch@15142f2c79
2015-06-12 21:38:02 +02:00
uboness 288ce368d5 Moved input errors to the input result
Until now, if the input failed, an exception would be thrown and it would be captured globally on the watch execution and in the watch recod message. The problem with this approach is that the information about the input is lost. In this commit, the failure is returned as part of the input result.

- A new `status` field was added to the input result. Can either have `success` or `failure` values. When set to `failure` a `reason` field will be set with the error message.
- The `ExecutionService` changed to enable this functionality. Mainly, instead of relying on exception, during the execution the input result is checked for its status and the execution is aborted on failure. Also, the two places where the watch execution is handled were consolidated to a single method `execute(WatchExecutionContext)`.
- Also, the watch execution context id (which will end up being the `watch_record` id) was added the the context model (accessible via scripts and templates). This is done mainly for debugging purposes.

Original commit: elastic/x-pack-elasticsearch@e2567deada
2015-06-12 14:08:07 +02:00
uboness b54b52bb2f Added version 1.0.0-Beta2
Original commit: elastic/x-pack-elasticsearch@ba6dfa6d96
2015-06-10 16:40:15 +02:00
uboness c5fd06d624 [test] added a test to make sure a failed action is not throttled
Closes elastic/elasticsearch#253

Original commit: elastic/x-pack-elasticsearch@89f9075731
2015-06-09 15:37:16 +02:00
uboness babe40137f [cleanup] Reorganized the integration tests
Distributed them to the proper packages, such that they'll be easy to find

Original commit: elastic/x-pack-elasticsearch@e6e6c15870
2015-06-09 14:19:47 +02:00
Martijn van Groningen 478d5ecc48 Stop the TriggeredWatchStore and HistoryStore if there are no current executions.
There may be current executions still going on during stopping.
We should try to wait for the current executions to have completed.
Otherwise we can run into a situation where we didn't delete the watch from the .triggered_watches index,
but did insert into the history index. Upon start this can lead to DocumentAlreadyExistsException,
because we already stored the history record during shutdown...

Introduced `CurrentExecutions` to handle this synchronization.

(we always first store the watch record and then remove the triggered watch)

Original commit: elastic/x-pack-elasticsearch@89c4a8d8ad
2015-06-08 20:38:37 +02:00
uboness e684194c23 Fix 1. compatibility
Original commit: elastic/x-pack-elasticsearch@e409f99754
2015-06-08 12:55:14 +02:00
uboness c491e4db16 Added support for multi doc indexing to index action
This commit adds support for indexing multiple documents with the `index` action. This is done by introducing a special `_doc` field. During action execution, the `_doc` field will be looked up in the payload. If found, the value of the field will be considered as the document that needs to be indexed. If the value is an array of objects, each object in that array will be treated as a separate document and all the documents in the array will be bulk indexed.

This commit also changes the result of the action to hold `XContentSource` rather than a payload (to avoid Map creation explosions). Th `XContentSource` was also extended to support lists.

Original commit: elastic/x-pack-elasticsearch@86f454b029
2015-06-07 17:17:38 +02:00
uboness ccdaf2f116 compare condition result
The `compare` condition result will now hold all resolved values, keyed by their associated parameter place holder strings.

Original commit: elastic/x-pack-elasticsearch@f930c77d54
2015-06-06 23:12:09 +02:00
uboness a9658cca77 Enable both timewarp on & off for ActionThrottleTests
Original commit: elastic/x-pack-elasticsearch@caa2da607f
2015-06-06 13:04:50 +02:00
uboness 096bd5b039 Fix throttling test failure
Original commit: elastic/x-pack-elasticsearch@f3799bddeb
2015-06-06 04:03:51 +02:00
uboness 034425185a Fixed failing test
Original commit: elastic/x-pack-elasticsearch@1291299318
2015-06-06 03:33:46 +02:00
Brian Murphy 8bf45f0340 [TEST] Watch action acking and throttling tests
This change adds tests to ack a subset of a watch's actions, use a different throttle period per action in a watch, also adds tests to make sure that both the watch level and global level throttle_period are applied correctly.
Also updates the REST tests to make sure that throttle periods can be set at a watch and action level and are returned from the GET API.

Original commit: elastic/x-pack-elasticsearch@4b006c7830
2015-06-06 02:35:42 +02:00
uboness d899c4b522 Execute API to support default trigger
Until today it was required to provide a trigger event as part of the execute API. There are two issues with it:

1. It's not user friendly (you'd expect that the execute API would just work with just pointing to the watch)
2. The API could expose inconsistencies where on one hand it points to a watch (with a well defined trigger) on the other it enabled the user to provide completely different trigger (of a different type)

This change enable supporting default triggers by enabling the trigger engine to create a simulated trigger event. This enables the execute API to look up the trigger type of the pointed watch, and ask the trigger service to simulate an event for it. It is still possible to override the trigger event data by providing it via the `trigger_data` parameter in the API.

This simplifies the execute API and prepares it for future trigger types as well.

- This commit add missing integration tests for the execute API
- Also, removed unused `setIgnoreThrottling` from the execute request/builder.

Original commit: elastic/x-pack-elasticsearch@b494ae62e6
2015-06-05 12:41:28 +02:00
Brian Murphy 2cff8c4c32 [TEST] Increase logging for long running delete test.
This test is failing on jenkins but does not reproduce.

Original commit: elastic/x-pack-elasticsearch@2f873261c0
2015-06-03 08:43:27 -04:00
Brian Murphy 14ffe290f7 Add support for http input endpoints that do not return JSON formatted bodys.
This change allows the httpinput to receive non json formatted data from a http endpoint
(such as the elasticsearch _cat apis). If non json is read it will be stored in the `payload._value` in the same
way that the `ScriptTransform` handles non map/json data returned by transforming scripts.
Added response_content_type to http input so that the expected content type in the response can be configured. This accepts `yaml`, `json` and `text` but will be overridden by the http headers.

Original commit: elastic/x-pack-elasticsearch@753d37f14e
2015-06-02 13:51:18 -04:00
uboness 613ce8762c De-normalize watch record format
Now that the watch record is write once and never read/parsed. We can de-normalize its format (the  structure of the `watch_record` document) such it'd be optimal for searching and aggregating.

- `execution_result` renamed to `result` (for minimal verbosity)

- the structure of `trigger_event` changed to:
 ```
 {
 	"trigger_event" : {
 		"type" : "<trigger_type>",
 		"triggered_time" : "<datetime>",
 		"<trigger_type>" : { // type specific data (optional) }
 	}
 }
 ```

- the structure of `input` results changed to:
 ```
 {
 	"result" : {
 		"input" : {
 			"type" : "<input_type>",
 			"payload" : { // the payload },
 			"<input_type>" : { // custom result fields per type (optional) }
 		},
 		...
 	}
 }
 ```

- the structure of `condition` results changed to:
 ```
 {
 	"result" : {
 	    ...
 		"condition" : {
 			"type" : "<condition_type>",
 			"met" : true | false,
 			"<condition_type>" : { // custom result fields per type (optional) }
 		},
 		...
 	}
 }
 ```

- the structure of `transform` results changed to:
 ```
 {
 	"result" : {
 	    ...
 		"transform" : {
 			"type" : "<transform_type>",
 			"payload" : { // the transformed payload }
 			"<transform_type>" : { // custom result fields per type (optional) }
 		},
 		...
 	}
 }
 ```

- the structure of `actions` results changed to:
 ```
 {
 	"result" : {
 	    ...
 		"actions" : [
 			{
 				"id" : "<action_id>"
 				"type" : "<action_type>",
 				"status" : "success" | "failure" | "simulated" | "throttled",
 				"reason" : "holds the reasoning if status is either success or throttled",
 				"transform" : { // action level transform result (if applicable)}
 				"<action_type>" : { // custom result fields per type (optional) }
 			},
 			...
 		]
 	}
 }
 ```

Original commit: elastic/x-pack-elasticsearch@98466d4b83
2015-06-02 03:57:21 +02:00
Martijn van Groningen 47247dc46a Stats api: fix serialization issue
Renamed pending watches left overs to queued watches

Original commit: elastic/x-pack-elasticsearch@c2bcdf547c
2015-05-31 13:14:32 +02:00
Martijn van Groningen 0890001470 stats: Add queued watches metric
The queued watches metric gives insight into the watches that are queued for execution.
Per watch that is queued, executing information is shared, like the `watch_id`,
when the watch was triggered and when execution started.

Original commit: elastic/x-pack-elasticsearch@deb5ddfde2
2015-05-31 11:13:13 +02:00
Martijn van Groningen 4335669635 Refactoring: Introduced TriggeredWatch concept.
Before WatchRecord was used to keep track of an execution of a Watch and used to store actual end results to it before sealing it. (for example build dashboard on the history indices)

The keeping track of an execution has been moved from WatchRecord to TriggeredWatch. If a watch triggers a TriggeredWatch is stored. The TriggeredWatch has its own index and only the watch_id (is part of id), trigger event and state is stored. If the execution of a Watch has finished (regardless if it was successfully) a WatchRecord is stored in a history index and the TriggeredWatch is deleted.

When a watch is getting executedtThe triggered watch is used the create the watch context.

Also the WatchRecord.State has been removed to its own enum class named ExecutionState. The CHECKING value has been removed, because it wasn't really used. The CHECKING state was set when the execution began, but it was never persisted and because of this state has also been removed from triggered watch.

By separating the result of a watch execution we are more flexible to in the future change the document format of WatchRecord. The history indices will be used by users to build analytics on top of watcher. Also the history indices become truely append only indices.

When update the watch status, only change the status part with the update api
Also set the version when we delete the watch on the in memory instance enforce more ensureStarted() in the components

Removed all watch record and result parsing code (actions, conditions, inputs and transforms)

Original commit: elastic/x-pack-elasticsearch@8f5ffdac13
2015-05-30 17:59:24 +02:00
uboness 6175b9efda Disallow negative time value settings
- Consolidated setting validation under `WatcherSettingsValidation`
- `WatcherSettingsException` is now only used for settings errors

We need this consolidation as Guice doesn't deal well with exceptions in constructors. So instead, `WatcherSettingsValidation` can be injected and used as a registry for settings errors and then, since it's a service, if there any registered errors, it'll throw `WatcherSettingsException` when it's started.

Fixes elastic/elasticsearch#539

Original commit: elastic/x-pack-elasticsearch@2c1895d18c
2015-05-28 23:24:18 +02:00
Martijn van Groningen f66f460313 test: removed @Seed
Original commit: elastic/x-pack-elasticsearch@aafc29f3b1
2015-05-28 12:40:33 +02:00
jaymode 9643ec2c62 add support for separate keystore and truststore in HttpClient
The HttpClient class tried to use the truststore as a keystore for watcher with Shield fallback.
This is trappy because in Shield, the default is to only have a keystore that also acts as the
truststore, so the fallback would probably fail in most cases. Additionally, support was missing
for a keystore that had a separate key password.

The new behavior allows for specifying both a keystore and a truststore. If only a keystore is
specified, it will also be used as the truststore. If neither is defined, the system truststore will
be used. All settings fallback to shield settings.

Closes elastic/elasticsearch#480

Original commit: elastic/x-pack-elasticsearch@ad02814806
2015-05-28 06:04:40 -04:00
uboness 45d0fdf3de Changed Execute Watch API response format
- the response now returns the id of the stored watch record
- update/fixed the docs

Closes elastic/elasticsearch#538

Original commit: elastic/x-pack-elasticsearch@5df7d166f4
2015-05-27 21:18:16 +02:00
uboness fb893e774a Add the ability to ack specific actions
- now it's possible to ack specific actions via the `Ack Watch API`
- Added tests for acking specific actions
- Changed the watch status structure such that the action ack state can be referred to by `status.actions.<action_id>.ack` (instead of `status.actions.<action_id>.ack_status`... removed the extra redundant "_status")
- As part of this work, also added validation for watch/action ids, such that we disallow having whitespaces in them.
- Updated the docs around acking & throttling of watch actions

Closes elastic/elasticsearch#531
Closes elastic/elasticsearch#537

Original commit: elastic/x-pack-elasticsearch@813e601bf5
2015-05-27 19:29:46 +02:00
uboness 6acc3f2616 Added support for headers in HttpClient
- HttpResponse now holds the response headers
- Added specific support for content type of the response, based on which we create the xcontent payload.

Original commit: elastic/x-pack-elasticsearch@beae27f576
2015-05-26 16:53:05 +02:00
Brian Murphy 46c111b016 Add execution_duration to watch history records.
This change add the actual length of time a watch spends executing. This is useful to find watches that take long to execute to pinpoint those watches that may be candidates for throttling.
Add the execution_duration as a number of milliseconds rather than a timevalue so it can be aggregated from the watch_history index.

Original commit: elastic/x-pack-elasticsearch@0036468f55
2015-05-26 10:34:24 -04:00
uboness 2f48d980b3 Fixing issues after master rebase
Original commit: elastic/x-pack-elasticsearch@d3dfa53483
2015-05-22 20:57:51 +02:00
uboness 33bc0761e7 fixed bad rebase merge
Original commit: elastic/x-pack-elasticsearch@a6793fedf8
2015-05-22 20:57:51 +02:00
uboness 575208c338 Centralized xcontent parsing of time values
- Renamed `WatcherDateUtils` to `WatcherDateTimeUtils`

Original commit: elastic/x-pack-elasticsearch@6b5557058a
2015-05-22 20:57:51 +02:00
uboness e0a70722e0 Move acking/throttling to the action level
Until now, acking and throttling functionality was applied at the watch level. This has major drawbacks in different aspects:

- When multiple actions are defined on a watch, acking a watch effectively acks all the actions. This is conceptually wrong. Say you have two actions: `email` and `index`. It's very likely you'd like to ack the email action (to avoid receiving too many emails) but at the same time continue indexing the data in the `index` action. Right now it's not possible.

- Different actions types may require different throttling. An `email` action probably needs a longer throttle period compared to an `index` action. Also for different `webhook` actions, the throttling is ultimately determined by the 3rd party system that is called.

This commit changes how we do throttling & acking. Moving this functionality to the action level. Now, when acking, each action in the watch will be acked separately. During executiong, each action will determine whether it needs to be throttled or not. The throttler is not associated with the action, not with the watch.

The throttle period was enhanced. There is a default throttle period that is configured for watcher as a whole (using the `watcher.execution.default_throttle_period` setting. Next to that, each `watch` can define its own `throttle_period` that can serve as the default throttle period for the actions in the watch. Lastly, each action can have its own throttle period set.

Since the throttler is now an action "thing", the `throttle` package was renamed to `throttler` and moved under the `actions` package. Also, `WatchThrottler` was renamed to `ActionThrottler`.

With this change, the `Watch Execute API` changed as well. Now, when executing a watch, you can define an execution mode per action. The execution mode offers 4 types of execution:
- `execute`: executes the watch normally (actually executing the action and it may be throttled)
- `force_execute`: skips/ignores throttling and executes the watch
- `simulate`: simulates the watch execution yet it may be throttled
- `force_simulate`: skips/ignores throttling and simulates the watch execution

As part of this change, the structure of the watch status changed along with the xconent representing the `watch_record`. A new `ActionStatus` was introduced (as part of the `WatchStatus`) and is always set for every action in the watch. This status holds:
 - the current state of the action (`ackable`, `awaits_successful_execution`, `acked`)
 - the last execution state (success/failure + reason)
 - the last successful execution state
 - the last throttle state (timestamp + reason)

Original commit: elastic/x-pack-elasticsearch@32c2985ed8
2015-05-22 20:57:51 +02:00
uboness ea91c1e617 third attempt to fix the failing tests windows
Original commit: elastic/x-pack-elasticsearch@30bb93ce4a
2015-05-22 11:39:09 +02:00