428 lines
17 KiB
Plaintext
428 lines
17 KiB
Plaintext
[[how-watcher-works]]
|
|
== How Watcher Works
|
|
|
|
Once you have <<getting-started, installed watcher>>, you can <<watch-definition, add watches>>
|
|
to automatically perform an action when certain conditions are met. The conditions are generally
|
|
based on data you've loaded into the watch by querying an Elasticsearch index or submitting an
|
|
HTTP request to a web service. For example, you could send an email to the sysadmin when a
|
|
search of your log data indicates that there are errors.
|
|
|
|
This topic describes the elements of a watch and how watches operate.
|
|
|
|
[[watch-definition]]
|
|
=== Watch Definition
|
|
|
|
A watch consists of a trigger, input, condition, and the actions you want to perform when the
|
|
watch condition is met. In addition, you can define transforms to process the watch payload
|
|
before executing the actions.
|
|
|
|
|
|
<<trigger,Trigger>> :: Determines when the watch is checked.
|
|
A watch must have a trigger.
|
|
|
|
<<input,Input>> :: Loads data into the watch payload.
|
|
If no input is specified, an empty payload is loaded.
|
|
|
|
<<condition,Condition>> :: Controls whether the watch actions are executed.
|
|
If no condition is specified, the condition defaults to `always`.
|
|
|
|
<<transform,Transform>> :: Processes the watch payload to prepare it for the watch actions.
|
|
You can define transforms at the watch level or define action-specific
|
|
transforms. Optional.
|
|
|
|
<<actions,Actions>> :: Specify what happens when the watch
|
|
condition is met.
|
|
|
|
[[watch-definition-example]]
|
|
|
|
For example, the following snippet shows a <<api-rest-put-watch, Put Watch>> request that defines
|
|
a watch that looks for log error events:
|
|
|
|
[source,json]
|
|
--------------------------------------------------
|
|
PUT _watcher/watch/log_event_watch
|
|
{
|
|
"metadata" : { <1>
|
|
"color" : "red"
|
|
},
|
|
"trigger" : { <2>
|
|
"schedule" : {
|
|
"interval" : "5m"
|
|
}
|
|
},
|
|
"input" : { <3>
|
|
"search" : {
|
|
"request" : {
|
|
"search_type" : "count",
|
|
"indices" : "log-events",
|
|
"body" : {
|
|
"query" : { "match" : { "status" : "error" } }
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"condition" : { <4>
|
|
"script" : "return ctx.payload.hits.total > 5"
|
|
},
|
|
"transform" : { <5>
|
|
"search" : {
|
|
"request" : {
|
|
"indices" : "log-events",
|
|
"body" : {
|
|
"query" : { "match" : { "status" : "error" } }
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"actions" : { <6>
|
|
"my_webhook" : {
|
|
"webhook" : {
|
|
"method" : "POST",
|
|
"host" : "mylisteninghost",
|
|
"port" : 9200,
|
|
"path" : "/{{watch_id}}",
|
|
"body" : "Encountered {{ctx.payload.hits.total}} errors"
|
|
}
|
|
},
|
|
"email_administrator" : {
|
|
"email" : {
|
|
"to" : "sys.admino@host.domain",
|
|
"subject" : "Encountered {{ctx.payload.hits.total}} errors",
|
|
"body" : "Too many error in the system, see attached data",
|
|
"attach_data" : true,
|
|
"priority" : "high"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// AUTOSENSE
|
|
|
|
<1> Metadata - You can attach optional static metadata to a watch.
|
|
<2> Trigger - This schedule trigger executes the watch every 5 minutes
|
|
<3> Input - This input searches for errors in the `log-events` index and loads the response
|
|
into the watch payload.
|
|
<4> Condition - This condition checks to see if there are more than 5 error events (hits in the
|
|
search response). If there are, execution continues.
|
|
<5> Transform - If the watch condition is met, this transform loads all of the errors into
|
|
the watch payload by searching for the errors using the default search type,
|
|
`query_then_fetch`. All of the watch actions have access to this payload.
|
|
<6> Actions - This watch has two actions. The `my_webhook` action notifies a 3rd party system
|
|
about the problem. The `email_administrator` action sends a high priority
|
|
email to the system administrator. The watch payload
|
|
that contains the errors is attached to the email.
|
|
|
|
[[watch-execution]]
|
|
=== Watch Execution
|
|
|
|
When you add a watch, Watcher immediately registers its trigger with the appropriate trigger
|
|
engine. Watches that have a `schedule` trigger are registered with the `scheduler` trigger engine.
|
|
The trigger engine is responsible for triggering execution of the watch. Trigger engines run on
|
|
the master node and use a separate thread pool from the one used to execute watches.
|
|
|
|
When a watch is triggered, Watcher queues it up for execution. A `watch_record` document is
|
|
created and added to the <<watch-history, watch history>> index and the initial status is set to
|
|
`awaits_execution`.
|
|
|
|
When execution starts, Watcher creates a watch execution context for the watch. The execution
|
|
context provides scripts and templates access to the watch metadata, payload, watch ID, execution
|
|
time, and trigger information. For more information, see
|
|
<<watch-execution-context, Watch Execution Context>>.
|
|
|
|
During the execution process, Watcher:
|
|
|
|
. Loads the input data into the payload in the watch execution context. This makes the data
|
|
available to all subsequent steps in the execution process. This step is controlled by the
|
|
input configured for the watch.
|
|
. Evaluates the watch condition to determine whether or not to continue processing the watch.
|
|
If the condition is met (evaluates to `true`), processing advances to the next step. If it
|
|
is not met (evaluates to `false`), execution of the watch stops.
|
|
. Applies transforms to the watch payload (if defined).
|
|
. Executes the watch actions if the condition is met and the watch is not being
|
|
<<watch-acknowledgment-throttling, throttled>>.
|
|
|
|
When watch execution finishes, Watcher updates the watch record with the execution results.
|
|
The watch record includes the execution time and duration, whether the watch condition was met,
|
|
and the status of each action that was performed. For more information, see
|
|
<<watch-history, Watch History>>.
|
|
|
|
The following diagram shows the watch execution process:
|
|
|
|
image::images/watch-execution.jpg[align="center"]
|
|
|
|
[[watch-acknowledgment-throttling]]
|
|
=== Watch Acknowledgment and Throttling
|
|
|
|
Watcher supports both time-based and acknowledgment-based throttling. This enables you to prevent
|
|
actions from being repeatedly executed for the same event.
|
|
|
|
By default, Watcher uses time-based throttling with a throttle period of 5 seconds. This means that
|
|
if a watch is executed every second, its actions are performed a maximum of once every 5 seconds,
|
|
even when the condition is met. You can configure the throttle period on a per-action basis, at the
|
|
watch level, or change the <<configuring-default-throttle-period, default throttle period>> in
|
|
`elasticsearch.yml`.
|
|
|
|
Acknowledgment-based throttling enables you to tell Watcher not to send any more notifications
|
|
about a watch as long as its condition remains true. Once the condition evaluates to `false`, the
|
|
acknowledgment is cleared and Watcher resumes executing the watch's actions normally.
|
|
|
|
For more information, see <<actions-ack-throttle, action throttling>>.
|
|
|
|
[[watch-active-state]]
|
|
=== Watch Active State
|
|
|
|
By default, when you add a watch it is immediately set to the active state. An active watch is
|
|
registered with the relevant trigger engine and executed according to its configured trigger.
|
|
For example, if a watch has a <<trigger-schedule, `schedule`>> trigger, it is executed according
|
|
to its schedule.
|
|
|
|
A watch can also be set to the inactive state. An inactive watch is still registered with Watcher,
|
|
but it is not registered with a trigger engine and can never be triggered. When you add a watch,
|
|
you can use the <<api-rest-put-watch-active-state, `active`>> parameter to set its initial state
|
|
to inactive. You can deactivate an existing watch with the <<api-rest-deactivate-watch, Deactivate Watch API>>.
|
|
To reactivate a watch, you use the <<api-rest-activate-watch, Activate Watch API>>.
|
|
|
|
NOTE: You can still use the <<api-rest-execute-watch, Execute Watch API>> to force the execution
|
|
of an inactive watch.
|
|
|
|
Deactivating watches is useful in a variety of situations. For example, if you have a watch that
|
|
monitors an external system and need to take the system down for maintenance, you can deactivate
|
|
the watch so doesn't unnecessarily report a bunch of execution failures during the maintenance
|
|
window. You might also want to deactivate a watch rather than deleting it entirely in case you
|
|
might want to use it at some point in the future.
|
|
|
|
|
|
[[scripts-templates]]
|
|
=== Scripts and Templates
|
|
|
|
You can use scripts and templates when defining a watch. Scripts and templates can reference
|
|
elements in the watch execution context, including the watch payload. The execution context defines
|
|
variables you can use in a script and parameter placeholders you can use in a template. Transforms
|
|
also update the contents of the watch payload.
|
|
|
|
Watcher uses the Elasticsearch script infrastructure, which supports <<inline-templates-scripts,inline>>,
|
|
<<indexed-templates-scripts, indexed>>, and <<file-templates-scripts, file-based scripts>>. Scripts
|
|
and templates are compiled and cached by Elasticsearch to optimize recurring execution.
|
|
{ref}/modules-scripting.html#_automatic_script_reloading[Autoloading] is also supported. For more
|
|
information, see {ref}/modules-scripting.html[Scripting] in the Elasticsearch Reference.
|
|
|
|
[[watch-execution-context]]
|
|
==== Watch Execution Context
|
|
|
|
The following snippet shows the basic elements in a watch's execution context:
|
|
|
|
[source,js]
|
|
----------------------------------------------------------------------
|
|
{
|
|
"ctx" : {
|
|
"metadata" : { ... }, <1>
|
|
"payload" : { ... }, <2>
|
|
"watch_id" : "<id>", <3>
|
|
"execution_time" : "20150220T00:00:10Z", <4>
|
|
"trigger" : { <5>
|
|
"triggered_time" : "20150220T00:00:10Z",
|
|
"scheduled_time" : "20150220T00:00:00Z"
|
|
},
|
|
"vars" : { ... } <6>
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
<1> Any static metadata specified in the watch definition.
|
|
<2> The current watch payload.
|
|
<3> The id of the executing watch.
|
|
<4> A timestamp that shows when the watch execution started.
|
|
<5> Information about the trigger event. For a `schedule` trigger, this
|
|
consists of the `triggered_time` (when the watch was triggered)
|
|
and the `scheduled_time` (when the watch was scheduled to be triggered).
|
|
<6> Dynamic variables that can be set and accessed by different constructs
|
|
during the execution. These variables are scoped to a single execution
|
|
(i.e they're not persisted and can't be used between different executions
|
|
of the same watch)
|
|
|
|
[[scripts]]
|
|
==== Using Scripts
|
|
|
|
You can use scripts to define <<condition-script, conditions>> and <<transform-script, transforms>>.
|
|
The default scripting language is groovy.
|
|
|
|
Scripts can reference any of the values in the watch execution context or values explicitly passed
|
|
through script parameters.
|
|
|
|
For example, if the context metadata contains a `color` field, `"metadata" : {"color": "red"}`, you
|
|
can access its value with the variable `ctx.metadata.color`. If you pass in a `color` parameter as
|
|
part of the condition or transform definition, `"params" : {"color": "red"}`, you access its value
|
|
with the variable `color`.
|
|
|
|
[[templates]]
|
|
==== Using Templates
|
|
|
|
You use templates to define dynamic content for a watch. At execution time, templates pull in data
|
|
from the watch's execution context. For example, you could use a template to populate the `subject`
|
|
field for an `email` action with data stored in the watch payload. Templates can also access values
|
|
explicitly passed through template parameters.
|
|
|
|
Watcher supports templates in a variety of places:
|
|
|
|
* The <<input-http, `http`>> input supports templates in the `path`, `params`, `headers` and
|
|
`body` fields.
|
|
* The <<actions-email, `email`>> action supports templates in the `from`, `reply_to`, `priority`,
|
|
`to`, `cc`, `bcc`, `subject`, `body.text` and `body.html` fields.
|
|
* The <<actions-webhook, `webhook`>> action supports templates in the `path`, `params`, `headers`
|
|
and `body` fields.
|
|
|
|
You specify templates using the https://mustache.github.io[Mustache] scripting language.
|
|
|
|
[NOTE]
|
|
===============================
|
|
While Elasticsearch supports Mustache out of the box, Watcher ships with its own version registered
|
|
under `xmustache`. This is because the default Mustache implementation in Elasticsearch 1.5 lacks
|
|
array/list access support. `xmustache` adds this support to enable easy array access. For example,
|
|
to refer to the source of the third search hit in the payload use
|
|
`{{ctx.payload.hits.hits.2._source}}`.
|
|
|
|
When this feature is available in Elasticsearch, we expect to remove `xmustache` from Watcher and
|
|
use the version that ships with Elasticsearch.
|
|
===============================
|
|
|
|
For example, if the context metadata contains a `color` field, you can access its value with the
|
|
expression `{{ctx.metadata.color}}`. If the context payload contains the results of a search, you
|
|
could access the source of the 3rd search hit in the payload with the following expression
|
|
`{{ctx.payload.hits.hits.2._source}}`.
|
|
|
|
If you pass in a parameter as part of the input or action definition, you can reference the
|
|
parameter by name. For example, the following snippet defines and references the `color` parameter.
|
|
|
|
[source,js]
|
|
----------------------------------------------------------------------
|
|
{
|
|
"actions" : {
|
|
"email_notification" : {
|
|
"email" : {
|
|
"subject" : {
|
|
"inline" : "{{color}} alert",
|
|
"params" : {
|
|
"color" : "red"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
[[inline-templates-scripts]]
|
|
==== Inline Templates and Scripts
|
|
|
|
To define an inline template or script, you simply specify it directly in the value of a field.
|
|
For example, the following snippet configures the subject of the `email` action using an inline
|
|
template that references the `color` value in the context metadata.
|
|
|
|
[source,js]
|
|
----------------------------------------------------------------------
|
|
"actions" : {
|
|
"email_notification" : {
|
|
"email" : {
|
|
"subject" : "{{ctx.metadata.color}} alert"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
For a script, you simply specify the inline script as the value of the `script` field.
|
|
For example:
|
|
|
|
[source,js]
|
|
----------------------------------------------------------------------
|
|
"condition" : {
|
|
"script" : "return true"
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
You can also explicitly specify the inline type by using a formal object definition as the field
|
|
value. For example:
|
|
|
|
[source,js]
|
|
----------------------------------------------------------------------
|
|
"actions" : {
|
|
"email_notification" : {
|
|
"email" : {
|
|
"subject" : {
|
|
"inline" : "{{ctx.metadata.color}} alert"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
The formal object definition for a script would be:
|
|
|
|
[source,js]
|
|
----------------------------------------------------------------------
|
|
"condition" : {
|
|
"script" : {
|
|
"inline": "return true"
|
|
}
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
[[indexed-templates-scripts]]
|
|
==== Indexed Templates and Scripts
|
|
|
|
If you {ref}/modules-scripting.html#_indexed_scripts[index] your templates and scripts, you can
|
|
reference them by id.
|
|
|
|
To reference an indexed script or template, you use the formal object definition and specify its
|
|
id in the `id` field. For example, the following snippet references the `email_notification_subject`
|
|
template.
|
|
|
|
[source,js]
|
|
----------------------------------------------------------------------
|
|
{
|
|
...
|
|
"actions" : {
|
|
"email_notification" : {
|
|
"email" : {
|
|
"subject" : {
|
|
"id" : "email_notification_subject",
|
|
"params" : {
|
|
"color" : "red"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
[[file-templates-scripts]]
|
|
==== File-based Templates and Scripts
|
|
|
|
If you store templates or scripts in the `$ES_HOME/config/scripts` directory, you can reference
|
|
them by name. Template files must be saved with the extension `.mustache`. Script files must be
|
|
saved with the appropriate file extension, such as `.groovy`.
|
|
|
|
NOTE: The `config/scripts` directory is scanned periodically for changes. New and changed
|
|
templates and scripts are reloaded and deleted templates and scripts are removed from
|
|
the preloaded scripts cache. For more information, see
|
|
{ref}/modules-scripting.html#_automatic_script_reloading[Automatic Script Reloading]
|
|
in the Elasticsearch Reference.
|
|
|
|
To reference a file-based index or script, you use the formal object definition and specify its
|
|
name in the `file` field. For example, the following snippet references the script file
|
|
`threshold_hits.groovy`.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"condition" : {
|
|
"script" : {
|
|
"file" : "threshold_hits",
|
|
"params" : {
|
|
"threshold" : 0
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
include::how-watcher-works/dynamic-index-names.asciidoc[]
|