OpenSearch/watcher/docs/how-watcher-works.asciidoc

[[how-watcher-works]]
== How Watcher Works

Once you have <<getting-started, installed watcher>>, you can <<watch-definition, add watches>>
to automatically perform an action when certain conditions are met. The conditions are generally
based on data you've loaded into the watch by querying an Elasticsearch index or submitting an
HTTP request to a web service. For example, you could send an email to the sysadmin when a
search of your log data indicates that there are errors. 

This topic describes the elements of a watch and how watches operate.

[[watch-definition]]
=== Watch Definition

A watch consists of a trigger, input, condition, and the actions you want to perform when the
watch condition is met. In addition, you can define transforms to process the watch payload
before executing the actions. 


<<trigger,Trigger>>     ::  Determines when the watch is checked. 
                            A watch must have a trigger.

<<input,Input>>         ::  Loads data into the watch payload.
                            If no input is specified, an empty payload is loaded.

<<condition,Condition>> ::  Controls whether the watch actions are executed.
                            If no condition is specified, the condition defaults to `always`.

<<transform,Transform>> ::  Processes the watch payload to prepare it for the watch actions. 
                            You can define transforms at the watch level or define action-specific
                            transforms. Optional.

<<actions,Actions>>     ::  Specify what happens when the watch
                            condition is met.

[[watch-definition-example]]

For example, the following snippet shows a <<api-rest-put-watch, Put Watch>> request that defines
a watch that looks for log error events:

[source,json]
--------------------------------------------------
PUT _watcher/watch/log_event_watch
{
  "metadata" : { <1>
    "color" : "red"
  },
  "trigger" : { <2>
    "schedule" : {
      "interval" : "5m"
    }
  },
  "input" : { <3>
    "search" : {
      "request" : {
        "indices" : "log-events",
        "body" : {
          "size" : 0,
          "query" : { "match" : { "status" : "error" } }
        }
      }
    }
  },
  "condition" : { <4>
    "script" : "return ctx.payload.hits.total > 5"
  },
  "transform" : { <5>
    "search" : {
        "request" : {
          "indices" : "log-events",
          "body" : {
            "query" : { "match" : { "status" : "error" } }
          }
        }
    }
  },
  "actions" : { <6>
    "my_webhook" : {
      "webhook" : {
        "method" : "POST",
        "host" : "mylisteninghost",
        "port" : 9200,
        "path" : "/{{watch_id}}",
        "body" : "Encountered {{ctx.payload.hits.total}} errors"
      }
    },
    "email_administrator" : {
      "email" : {
        "to" : "sys.admino@host.domain",
        "subject" : "Encountered {{ctx.payload.hits.total}} errors",
        "body" : "Too many error in the system, see attached data",
        "attach_data" : true,
        "priority" : "high"
      }
    }
  }
}
--------------------------------------------------
// AUTOSENSE

<1> Metadata  - You can attach optional static metadata to a watch.
<2> Trigger   - This schedule trigger executes the watch every 5 minutes
<3> Input     - This input searches for errors in the `log-events` index and loads the response
                into the watch payload.
<4> Condition - This condition checks to see if there are more than 5 error events (hits in the
                search response). If there are, execution continues.
<5> Transform - If the watch condition is met, this transform loads all of the errors into 
                the watch payload by searching for the errors using the default search type, 
                `query_then_fetch`. All of the watch actions have access to this payload.
<6> Actions   - This watch has two actions. The `my_webhook` action notifies a 3rd party system
                about the problem. The `email_administrator` action sends a high priority
                email to the system administrator. The watch payload
                that contains the errors is attached to the email.

[[watch-execution]]
=== Watch Execution

When you add a watch, Watcher immediately registers its trigger with the appropriate trigger
engine. Watches that have a `schedule` trigger are registered with the `scheduler` trigger engine.
The trigger engine is responsible for triggering execution of the watch. Trigger engines run on
the master node and use a separate thread pool from the one used to execute watches.

When a watch is triggered, Watcher queues it up for execution. A `watch_record` document is
created and added to the <<watch-history, watch history>> index and the initial status is set to
`awaits_execution`.

When execution starts, Watcher creates a watch execution context for the watch. The execution
context provides scripts and templates access to the watch metadata, payload, watch ID, execution
time, and trigger information. For more information, see
<<watch-execution-context, Watch Execution Context>>.

During the execution process, Watcher:

. Loads the input data into the payload in the watch execution context. This makes the data
  available to all subsequent steps in the execution process. This step is controlled by the
  input configured for the watch.
. Evaluates the watch condition to determine whether or not to continue processing the watch.
  If the condition is met (evaluates to `true`), processing advances to the next step. If it
  is not met (evaluates to `false`), execution of the watch stops.
. Applies transforms to the watch payload (if defined).
. Executes the watch actions if the condition is met and the watch is not being
  <<watch-acknowledgment-throttling, throttled>>.

When watch execution finishes, Watcher updates the watch record with the execution results.
The watch record includes the execution time and duration, whether the watch condition was met, 
and the status of each action that was performed. For more information, see
<<watch-history, Watch History>>.

The following diagram shows the watch execution process:

image::images/watch-execution.jpg[align="center"]

[[watch-acknowledgment-throttling]]
=== Watch Acknowledgment and Throttling

Watcher supports both time-based and acknowledgment-based throttling. This enables you to prevent
actions from being repeatedly executed for the same event. 

By default, Watcher uses time-based throttling with a throttle period of 5 seconds. This means that
if a watch is executed every second, its actions are performed a maximum of once every 5 seconds,
even when the condition is met. You can configure the throttle period on a per-action basis, at the
watch level, or  change the  <<configuring-default-throttle-period, default throttle period>> in 
`elasticsearch.yml`.

Acknowledgment-based throttling enables you to tell Watcher not to send any more notifications
about a watch as long as its condition remains true. Once the condition evaluates to `false`, the
acknowledgment is cleared and Watcher resumes executing the watch's actions normally. 

For more information, see <<actions-ack-throttle, action throttling>>.

[[watch-active-state]]
=== Watch Active State

By default, when you add a watch it is immediately set to the active state. An active watch is
registered with the relevant trigger engine and executed according to its configured trigger.
For example, if a watch has a <<trigger-schedule, `schedule`>> trigger, it is executed according
to its schedule.

A watch can also be set to the inactive state. An inactive watch is still registered with Watcher,
but it is not registered with a trigger engine and can never be triggered. When you add a watch,
you can use the <<api-rest-put-watch-active-state, `active`>> parameter to set its initial state
to inactive. You can deactivate an existing watch with the <<api-rest-deactivate-watch, Deactivate Watch API>>.
To reactivate a watch, you use the <<api-rest-activate-watch, Activate Watch API>>.

NOTE:   You can still use the <<api-rest-execute-watch, Execute Watch API>> to force the execution
        of an inactive watch.

Deactivating watches is useful in a variety of situations. For example, if you have a watch that
monitors an external system and need to take the system down for maintenance, you can deactivate
the watch so doesn't unnecessarily report a bunch of execution failures during the maintenance
window. You might also want to deactivate a watch rather than deleting it entirely in case you
might want to use it at some point in the future.


[[scripts-templates]]
=== Scripts and Templates

You can use scripts and templates when defining a watch. Scripts and templates can reference
elements in the watch execution context, including the watch payload. The execution context defines
variables you can use in a script and parameter placeholders you can use in a template. Transforms
also update the contents of the watch payload.

Watcher uses the Elasticsearch script infrastructure, which supports <<inline-templates-scripts,inline>>,
<<indexed-templates-scripts, indexed>>, and <<file-templates-scripts, file-based scripts>>. Scripts
and templates are compiled and cached by Elasticsearch to optimize recurring execution.
{ref}/modules-scripting.html#_automatic_script_reloading[Autoloading] is also supported. For more
information, see {ref}/modules-scripting.html[Scripting] in the Elasticsearch Reference.

[[watch-execution-context]]
==== Watch Execution Context

The following snippet shows the basic elements in a watch's execution context:

[source,js]
----------------------------------------------------------------------
{
  "ctx" : {
    "metadata" : { ... }, <1>
    "payload" : { ... }, <2>
    "watch_id" : "<id>", <3>
    "execution_time" : "20150220T00:00:10Z", <4>
    "trigger" : { <5>
      "triggered_time" : "20150220T00:00:10Z",
      "scheduled_time" : "20150220T00:00:00Z"
    },
    "vars" : { ... } <6>
}
----------------------------------------------------------------------

<1> Any static metadata specified in the watch definition.
<2> The current watch payload.
<3> The id of the executing watch.
<4> A timestamp that shows when the watch execution started.
<5> Information about the trigger event. For a `schedule` trigger, this
    consists of the `triggered_time` (when the watch was triggered) 
    and the `scheduled_time` (when the watch was scheduled to be triggered).
<6> Dynamic variables that can be set and accessed by different constructs
    during the execution. These variables are scoped to a single execution
    (i.e they're not persisted and can't be used between different executions
    of the same watch)

[[scripts]]
==== Using Scripts

You can use scripts to define <<condition-script, conditions>> and <<transform-script, transforms>>.
The default scripting language is groovy.

Scripts can reference any of the values in the watch execution context or values explicitly passed
through script parameters. 

For example, if the context metadata contains a `color` field, `"metadata" : {"color": "red"}`, you
can access its value with the variable `ctx.metadata.color`. If you pass in a `color`  parameter as
part of the condition or transform definition, `"params" : {"color": "red"}`, you access its value
with the variable `color`.

[[templates]]
==== Using Templates

You use templates to define dynamic content for a watch. At execution time, templates pull in data
from the watch's execution context. For example, you could use a template to populate the `subject`
field for an `email` action with data stored in the watch payload. Templates can also access values
explicitly passed through template parameters. 

Watcher supports templates in a variety of places:

* The <<input-http, `http`>> input supports templates in the `path`, `params`, `headers` and
  `body` fields.
* The <<actions-email, `email`>> action supports templates in the `from`, `reply_to`, `priority`,
  `to`, `cc`, `bcc`, `subject`, `body.text` and `body.html` fields.
* The <<actions-webhook, `webhook`>> action supports templates in the `path`, `params`, `headers`
  and `body` fields.

You specify templates using the https://mustache.github.io[Mustache] scripting language. 

[NOTE]
===============================
While Elasticsearch supports Mustache out of the box, Watcher ships with its own version registered
under `xmustache`. This is because the default Mustache implementation in Elasticsearch 1.5 lacks
array/list access support. `xmustache` adds this support to enable easy array access. For example,
to refer to the source of the third search hit in the payload use
`{{ctx.payload.hits.hits.2._source}}`.

When this feature is available in Elasticsearch, we expect to remove `xmustache` from Watcher and
use the version that ships with Elasticsearch.
===============================

For example, if the context metadata contains a `color` field, you can access its value with the
expression `{{ctx.metadata.color}}`. If the context payload contains the results of a search, you
could access the source of the 3rd search hit in the payload with the following expression
`{{ctx.payload.hits.hits.2._source}}`.

If you pass in a parameter as part of the input or action definition, you can reference the
parameter by name. For example, the following snippet defines and references the `color` parameter.

[source,js]
----------------------------------------------------------------------
{
  "actions" : {
    "email_notification" : {
      "email" : {
        "subject" : {
          "inline" : "{{color}} alert",
          "params" : {
            "color" : "red"
          }
        }
      }
    }
  }
}
----------------------------------------------------------------------

[[inline-templates-scripts]]
==== Inline Templates and Scripts

To define an inline template or script, you simply specify it directly in the value of a field.
For example, the following snippet configures the subject of the `email` action using an inline
template that references the `color` value in the context metadata. 

[source,js]
----------------------------------------------------------------------
"actions" : {
  "email_notification" : {
     "email" : {
       "subject" : "{{ctx.metadata.color}} alert"
     }
   }
  }
}
----------------------------------------------------------------------

For a script, you simply specify the inline script as the value of the `script` field.
For example:

[source,js]
----------------------------------------------------------------------
"condition" : {
  "script" : "return true"
}
----------------------------------------------------------------------

You can also explicitly specify the inline type by using a formal object definition as the field
value. For example:

[source,js]
----------------------------------------------------------------------
"actions" : {
  "email_notification" : {
    "email" : {
      "subject" : {
         "inline" : "{{ctx.metadata.color}} alert"
      }
    }
  }
}
----------------------------------------------------------------------

The formal object definition for a script would be:

[source,js]
----------------------------------------------------------------------
"condition" : {
  "script" : {
    "inline": "return true"
  }
}
----------------------------------------------------------------------

[[indexed-templates-scripts]]
==== Indexed Templates and Scripts

If you {ref}/modules-scripting.html#_indexed_scripts[index] your templates and scripts, you can
reference them by id. 

To reference an indexed script or template, you use the formal object definition and specify its
id in the `id` field. For example, the following snippet references the `email_notification_subject`
template.

[source,js]
----------------------------------------------------------------------
{
  ...
  "actions" : {
    "email_notification" : {
      "email" : {
        "subject" : {
          "id" : "email_notification_subject",
          "params" : {
            "color" : "red"
          }
        }
      }
    }
  }
}
----------------------------------------------------------------------

[[file-templates-scripts]]
==== File-based Templates and Scripts

If you store templates or scripts in the `$ES_HOME/config/scripts` directory, you can reference
them by name. Template files must be saved with the extension `.mustache`. Script files must be
saved with the appropriate file extension, such as `.groovy`.

NOTE:   The `config/scripts` directory is scanned periodically for changes. New and changed
        templates and scripts are reloaded and deleted templates and scripts are removed from
        the preloaded scripts cache. For more information, see 
        {ref}/modules-scripting.html#_automatic_script_reloading[Automatic Script Reloading]
        in the Elasticsearch Reference.

To reference a file-based index or script, you use the formal object definition and specify its
name in the `file` field. For example, the following snippet references the script file
`threshold_hits.groovy`.

[source,js]
--------------------------------------------------
"condition" : {
    "script" : {
      "file" : "threshold_hits",
      "params" : {
        "threshold" : 0 
      }
    }
  }
--------------------------------------------------