[[how-watcher-works]] == How Watcher Works Once you have <>, you can <> to automatically perform an action when certain conditions are met. The conditions are generally based on data you've loaded into the watch by querying an Elasticsearch index or submitting an HTTP request to a web service. For example, you could send an email to the sysadmin when a search of your log data indicates that there are errors. This topic describes the elements of a watch and how watches operate. [[watch-definition]] === Watch Definition A watch consists of a trigger, input, condition, and the actions you want to perform when the watch condition is met. In addition, you can define transforms to process the watch payload before executing the actions. <> :: Determines when the watch is checked. A watch must have a trigger. <> :: Loads data into the watch payload. If no input is specified, an empty payload is loaded. <> :: Controls whether the watch actions are executed. If no condition is specified, the condition defaults to `always`. <> :: Processes the watch payload to prepare it for the watch actions. You can define transforms at the watch level or define action-specific transforms. Optional. <> :: Specify what happens when the watch condition is met. [[watch-definition-example]] For example, the following snippet shows a <> request that defines a watch that looks for log error events: [source,json] -------------------------------------------------- PUT _watcher/watch/log_event_watch { "metadata" : { <1> "color" : "red" }, "trigger" : { <2> "schedule" : { "interval" : "5m" } }, "input" : { <3> "search" : { "request" : { "search_type" : "count", "indices" : "log-events", "body" : { "query" : { "match" : { "status" : "error" } } } } } }, "condition" : { <4> "script" : "return ctx.payload.hits.total > 5" }, "transform" : { <5> "search" : { "request" : { "indices" : "log-events", "body" : { "query" : { "match" : { "status" : "error" } } } } } }, "actions" : { <6> "my_webhook" : { "webhook" : { "method" : "POST", "host" : "mylisteninghost", "port" : 9200, "path" : "/{{watch_id}}", "body" : "Encountered {{ctx.payload.hits.total}} errors" } }, "email_administrator" : { "email" : { "to" : "sys.admino@host.domain", "subject" : "Encountered {{ctx.payload.hits.total}} errors", "body" : "Too many error in the system, see attached data", "attach_data" : true, "priority" : "high" } } } } -------------------------------------------------- // AUTOSENSE <1> Metadata - You can attach optional static metadata to a watch. <2> Trigger - This schedule trigger executes the watch every 5 minutes <3> Input - This input searches for errors in the `log-events` index and loads the response into the watch payload. <4> Condition - This condition checks to see if there are more than 5 error events (hits in the search response). If there are, execution continues. <5> Transform - If the watch condition is met, this transform loads all of the errors into the watch payload by searching for the errors using the default search type, `query_then_fetch`. All of the watch actions have access to this payload. <6> Actions - This watch has two actions. The `my_webhook` action notifies a 3rd party system about the problem. The `email_administrator` action sends a high priority email to the system administrator. The watch payload that contains the errors is attached to the email. [[watch-execution]] === Watch Execution When you add a watch, Watcher immediately registers its trigger with the appropriate trigger engine. Watches that have a `schedule` trigger are registered with the `scheduler` trigger engine. The trigger engine is responsible for triggering execution of the watch. Trigger engines run on the master node and use a separate thread pool from the one used to execute watches. When a watch is triggered, Watcher queues it up for execution. A `watch_record` document is created and added to the <> index and the initial status is set to `awaits_execution`. When execution starts, Watcher creates a watch execution context for the watch. The execution context provides scripts and templates access to the watch metadata, payload, watch ID, execution time, and trigger information. For more information, see <>. During the execution process, Watcher: . Loads the input data into the payload in the watch execution context. This makes the data available to all subsequent steps in the execution process. This step is controlled by the input configured for the watch. . Evaluates the watch condition to determine whether or not to continue processing the watch. If the condition is met (evaluates to `true`), processing advances to the next step. If it is not met (evaluates to `false`), execution of the watch stops. . Applies transforms to the watch payload (if defined). . Executes the watch actions if the condition is met and the watch is not being <>. When watch execution finishes, Watcher updates the watch record with the execution results. The watch record includes the execution time and duration, whether the watch condition was met, and the status of each action that was performed. For more information, see <>. The following diagram shows the watch execution process: image::images/watch-execution.jpg[align="center"] [[watch-acknowledgment-throttling]] === Watch Acknowledgment and Throttling Watcher supports both time-based and acknowledgment-based throttling. This enables you to prevent actions from being repeatedly executed for the same event. By default, Watcher uses time-based throttling with a throttle period of 5 seconds. This means that if a watch is executed every second, its actions are performed a maximum of once every 5 seconds, even when the condition is met. You can configure the throttle period on a per-action basis, at the watch level, or change the <> in `elasticsearch.yml`. Acknowledgment-based throttling enables you to tell Watcher not to send any more notifications about a watch as long as its condition remains true. Once the condition evaluates to `false`, the acknowledgment is cleared and Watcher resumes executing the watch's actions normally. For more information, see <>. [[scripts-templates]] === Scripts and Templates You can use scripts and templates when defining a watch. Scripts and templates can reference elements in the watch execution context, including the watch payload. The execution context defines variables you can use in a script and parameter placeholders you can use in a template. Transforms also update the contents of the watch payload. Watcher uses the Elasticsearch script infrastructure, which supports <>, <>, and <>. Scripts and templates are compiled and cached by Elasticsearch to optimize recurring execution. {ref}/modules-scripting.html#_automatic_script_reloading[Autoloading] is also supported. For more information, see {ref}/modules-scripting.html[Scripting] in the Elasticsearch Reference. [[watch-execution-context]] ==== Watch Execution Context The following snippet shows the basic elements in a watch's execution context: [source,js] ---------------------------------------------------------------------- { "ctx" : { "metadata" : { ... }, <1> "payload" : { ... }, <2> "watch_id" : "", <3> "execution_time" : "20150220T00:00:10Z", <4> "trigger" : { <5> "triggered_time" : "20150220T00:00:10Z", "scheduled_time" : "20150220T00:00:00Z" }, "vars" : { ... } <6> } ---------------------------------------------------------------------- <1> Any static metadata specified in the watch definition. <2> The current watch payload. <3> The id of the executing watch. <4> A timestamp that shows when the watch execution started. <5> Information about the trigger event. For a `schedule` trigger, this consists of the `triggered_time` (when the watch was triggered) and the `scheduled_time` (when the watch was scheduled to be triggered). <6> Dynamic variables that can be set and accessed by different constructs during the execution. These variables are scoped to a single execution (i.e they're not persisted and can't be used between different executions of the same watch) [[scripts]] ==== Using Scripts You can use scripts to define <> and <>. The default scripting language is groovy. Scripts can reference any of the values in the watch execution context or values explicitly passed through script parameters. For example, if the context metadata contains a `color` field, `"metadata" : {"color": "red"}`, you can access its value with the variable `ctx.metadata.color`. If you pass in a `color` parameter as part of the condition or transform definition, `"params" : {"color": "red"}`, you access its value with the variable `color`. [[templates]] ==== Using Templates You use templates to define dynamic content for a watch. At execution time, templates pull in data from the watch's execution context. For example, you could use a template to populate the `subject` field for an `email` action with data stored in the watch payload. Templates can also access values explicitly passed through template parameters. Watcher supports templates in a variety of places: * The <> input supports templates in the `path`, `params`, `headers` and `body` fields. * The <> action supports templates in the `from`, `reply_to`, `priority`, `to`, `cc`, `bcc`, `subject`, `body.text` and `body.html` fields. * The <> action supports templates in the `path`, `params`, `headers` and `body` fields. You specify templates using the https://mustache.github.io[Mustache] scripting language. [NOTE] =============================== While Elasticsearch supports Mustache out of the box, Watcher ships with its own version registered under `xmustache`. This is because the default Mustache implementation in Elasticsearch 1.5 lacks array/list access support. `xmustache` adds this support to enable easy array access. For example, to refer to the source of the third search hit in the payload use `{{ctx.payload.hits.hits.2._source}}`. When this feature is available in Elasticsearch, we expect to remove `xmustache` from Watcher and use the version that ships with Elasticsearch. =============================== For example, if the context metadata contains a `color` field, you can access its value with the expression `{{ctx.metadata.color}}`. If the context payload contains the results of a search, you could access the source of the 3rd search hit in the payload with the following expression `{{ctx.payload.hits.hits.2._source}}`. If you pass in a parameter as part of the input or action definition, you can reference the parameter by name. For example, the following snippet defines and references the `color` parameter. [source,js] ---------------------------------------------------------------------- { "actions" : { "email_notification" : { "email" : { "subject" : { "inline" : "{{color}} alert", "params" : { "color" : "red" } } } } } } ---------------------------------------------------------------------- [[inline-templates-scripts]] ==== Inline Templates and Scripts To define an inline template or script, you simply specify it directly in the value of a field. For example, the following snippet configures the subject of the `email` action using an inline template that references the `color` value in the context metadata. [source,js] ---------------------------------------------------------------------- "actions" : { "email_notification" : { "email" : { "subject" : "{{ctx.metadata.color}} alert" } } } } ---------------------------------------------------------------------- For a script, you simply specify the inline script as the value of the `script` field. For example: [source,js] ---------------------------------------------------------------------- "condition" : { "script" : "return true" } ---------------------------------------------------------------------- You can also explicitly specify the inline type by using a formal object definition as the field value. For example: [source,js] ---------------------------------------------------------------------- "actions" : { "email_notification" : { "email" : { "subject" : { "inline" : "{{ctx.metadata.color}} alert" } } } } ---------------------------------------------------------------------- The formal object definition for a script would be: [source,js] ---------------------------------------------------------------------- "condition" : { "script" : { "inline": "return true" } } ---------------------------------------------------------------------- [[indexed-templates-scripts]] ==== Indexed Templates and Scripts If you {ref}/modules-scripting.html#_indexed_scripts[index] your templates and scripts, you can reference them by id. To reference an indexed script or template, you use the formal object definition and specify its id in the `id` field. For example, the following snippet references the `email_notification_subject` template. [source,js] ---------------------------------------------------------------------- { ... "actions" : { "email_notification" : { "email" : { "subject" : { "id" : "email_notification_subject", "params" : { "color" : "red" } } } } } } ---------------------------------------------------------------------- [[file-templates-scripts]] ==== File-based Templates and Scripts If you store templates or scripts in the `$ES_HOME/config/scripts` directory, you can reference them by name. Template files must be saved with the extension `.mustache`. Script files must be saved with the appropriate file extension, such as `.groovy`. NOTE: The `config/scripts` directory is scanned periodically for changes. New and changed templates and scripts are reloaded and deleted templates and scripts are removed from the preloaded scripts cache. For more information, see {ref}/modules-scripting.html#_automatic_script_reloading[Automatic Script Reloading] in the Elasticsearch Reference. To reference a file-based index or script, you use the formal object definition and specify its name in the `file` field. For example, the following snippet references the script file `threshold_hits.groovy`. [source,js] -------------------------------------------------- "condition" : { "script" : { "file" : "threshold_hits", "params" : { "threshold" : 0 } } } -------------------------------------------------- include::how-watcher-works/dynamic-index-names.asciidoc[]