2019-09-10 13:32:51 -04:00
|
|
|
[role="xpack"]
|
2017-03-28 17:23:01 -04:00
|
|
|
[[watcher-getting-started]]
|
2019-09-10 13:32:51 -04:00
|
|
|
== Getting started with {watcher}
|
2017-03-28 17:23:01 -04:00
|
|
|
|
2019-09-10 13:32:51 -04:00
|
|
|
TIP: To complete these steps, you must obtain a license that includes the
|
|
|
|
{alert-features}. For more information about Elastic license levels, see
|
2019-09-30 13:18:50 -04:00
|
|
|
https://www.elastic.co/subscriptions and
|
2020-06-16 19:43:54 -04:00
|
|
|
{kibana-ref}/managing-licenses.html[License management].
|
2017-03-28 17:23:01 -04:00
|
|
|
|
|
|
|
[[watch-log-data]]
|
|
|
|
To set up a watch to start sending alerts:
|
|
|
|
|
|
|
|
* <<log-add-input, Schedule the watch and define an input>>.
|
|
|
|
* <<log-add-condition, Add a condition>> that checks to see if an alert
|
|
|
|
needs to be sent.
|
|
|
|
* <<log-take-action, Configure an action>> to send an alert when the
|
|
|
|
condition is met.
|
|
|
|
|
|
|
|
[float]
|
|
|
|
[[log-add-input]]
|
2019-09-30 13:18:50 -04:00
|
|
|
=== Schedule the watch and define an input
|
2017-03-28 17:23:01 -04:00
|
|
|
|
2019-09-30 13:18:50 -04:00
|
|
|
A watch <<trigger-schedule,schedule>> controls how often a watch is triggered.
|
|
|
|
The watch <<input,input>> gets the data that you want to evaluate.
|
2017-03-28 17:23:01 -04:00
|
|
|
|
|
|
|
To periodically search log data and load the results into the
|
2019-09-30 13:18:50 -04:00
|
|
|
watch, you could use an <<schedule-interval,interval>> schedule and a
|
|
|
|
<<input-search,search>> input. For example, the following Watch searches
|
2017-03-28 17:23:01 -04:00
|
|
|
the `logs` index for errors every 10 seconds:
|
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2017-03-28 17:23:01 -04:00
|
|
|
------------------------------------------------------------
|
2018-12-08 13:57:16 -05:00
|
|
|
PUT _watcher/watch/log_error_watch
|
2017-03-28 17:23:01 -04:00
|
|
|
{
|
|
|
|
"trigger" : {
|
|
|
|
"schedule" : { "interval" : "10s" } <1>
|
|
|
|
},
|
|
|
|
"input" : {
|
|
|
|
"search" : {
|
|
|
|
"request" : {
|
|
|
|
"indices" : [ "logs" ],
|
|
|
|
"body" : {
|
|
|
|
"query" : {
|
|
|
|
"match" : { "message": "error" }
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
------------------------------------------------------------
|
2019-09-09 12:35:50 -04:00
|
|
|
|
2017-03-28 17:23:01 -04:00
|
|
|
<1> Schedules are typically configured to run less frequently. This example sets
|
|
|
|
the interval to 10 seconds so you can easily see the watches being triggered.
|
2017-07-20 00:24:29 -04:00
|
|
|
Since this watch runs so frequently, don't forget to <<log-delete, delete the watch>>
|
2017-03-28 17:23:01 -04:00
|
|
|
when you're done experimenting.
|
|
|
|
|
|
|
|
If you check the watch history you'll see that the watch is being triggered every
|
|
|
|
10 seconds. However, the search isn't returning any results so nothing is loaded
|
|
|
|
into the watch payload.
|
|
|
|
|
|
|
|
For example, the following request retrieves the last ten watch executions (watch
|
|
|
|
records) from the watch history:
|
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2017-03-28 17:23:01 -04:00
|
|
|
------------------------------------------------------------
|
|
|
|
GET .watcher-history*/_search?pretty
|
|
|
|
{
|
|
|
|
"sort" : [
|
|
|
|
{ "result.execution_time" : "desc" }
|
|
|
|
]
|
|
|
|
}
|
|
|
|
------------------------------------------------------------
|
|
|
|
// TEST[continued]
|
|
|
|
|
|
|
|
[float]
|
|
|
|
[[log-add-condition]]
|
2019-09-30 13:18:50 -04:00
|
|
|
=== Add a condition
|
2017-03-28 17:23:01 -04:00
|
|
|
|
2019-09-30 13:18:50 -04:00
|
|
|
A <<condition,condition>> evaluates the data you've loaded into the watch and
|
2017-03-28 17:23:01 -04:00
|
|
|
determines if any action is required. Now that you've loaded log errors into
|
|
|
|
the watch, you can define a condition that checks to see if any errors were
|
|
|
|
found.
|
|
|
|
|
|
|
|
For example, the following compare condition simply checks to see if the
|
|
|
|
search input returned any hits.
|
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2017-03-28 17:23:01 -04:00
|
|
|
--------------------------------------------------
|
2018-12-08 13:57:16 -05:00
|
|
|
PUT _watcher/watch/log_error_watch
|
2017-03-28 17:23:01 -04:00
|
|
|
{
|
|
|
|
"trigger" : { "schedule" : { "interval" : "10s" }},
|
|
|
|
"input" : {
|
|
|
|
"search" : {
|
|
|
|
"request" : {
|
|
|
|
"indices" : [ "logs" ],
|
|
|
|
"body" : {
|
|
|
|
"query" : {
|
|
|
|
"match" : { "message": "error" }
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
},
|
|
|
|
"condition" : {
|
2018-12-05 13:49:06 -05:00
|
|
|
"compare" : { "ctx.payload.hits.total.value" : { "gt" : 0 }} <1>
|
2017-03-28 17:23:01 -04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
2019-09-09 12:35:50 -04:00
|
|
|
|
2019-09-30 13:18:50 -04:00
|
|
|
<1> The <<condition-compare,compare>> condition lets you easily compare against
|
2017-03-28 17:23:01 -04:00
|
|
|
values in the execution context.
|
|
|
|
|
|
|
|
For this compare condition to evaluate to `true`, you need to add an event
|
|
|
|
to the `logs` index that contains an error. For example, the following request
|
|
|
|
adds a 404 error to the `logs` index:
|
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2017-03-28 17:23:01 -04:00
|
|
|
--------------------------------------------------
|
|
|
|
POST logs/event
|
|
|
|
{
|
|
|
|
"timestamp" : "2015-05-17T18:12:07.613Z",
|
|
|
|
"request" : "GET index.html",
|
|
|
|
"status_code" : 404,
|
|
|
|
"message" : "Error: File not found"
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
// TEST[continued]
|
|
|
|
|
|
|
|
Once you add this event, the next time the watch executes its condition will
|
2017-06-27 20:16:51 -04:00
|
|
|
evaluate to `true`. The condition result is recorded as part of the
|
2017-03-28 17:23:01 -04:00
|
|
|
`watch_record` each time the watch executes, so you can verify whether or
|
|
|
|
not the condition was met by searching the watch history:
|
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2017-03-28 17:23:01 -04:00
|
|
|
--------------------------------------------------
|
|
|
|
GET .watcher-history*/_search?pretty
|
|
|
|
{
|
|
|
|
"query" : {
|
|
|
|
"bool" : {
|
|
|
|
"must" : [
|
|
|
|
{ "match" : { "result.condition.met" : true }},
|
|
|
|
{ "range" : { "result.execution_time" : { "from" : "now-10s" }}}
|
|
|
|
]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
// TEST[continued]
|
|
|
|
|
|
|
|
[float]
|
|
|
|
[[log-take-action]]
|
2019-09-30 13:18:50 -04:00
|
|
|
=== Configure an action
|
2017-03-28 17:23:01 -04:00
|
|
|
|
|
|
|
Recording watch records in the watch history is nice, but the real power of
|
|
|
|
{watcher} is being able to do something when the watch condition is met. A
|
2019-09-30 13:18:50 -04:00
|
|
|
watch's <<actions,actions>> define what to do when the watch condition
|
2017-03-28 17:23:01 -04:00
|
|
|
evaluates to `true`. You can send emails, call third-party webhooks, write
|
|
|
|
documents to an Elasticsearch index, or log messages to the standard
|
|
|
|
Elasticsearch log files.
|
|
|
|
|
|
|
|
For example, the following action writes a message to the Elasticsearch
|
|
|
|
log when an error is detected.
|
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2017-03-28 17:23:01 -04:00
|
|
|
--------------------------------------------------
|
2018-12-08 13:57:16 -05:00
|
|
|
PUT _watcher/watch/log_error_watch
|
2017-03-28 17:23:01 -04:00
|
|
|
{
|
|
|
|
"trigger" : { "schedule" : { "interval" : "10s" }},
|
|
|
|
"input" : {
|
|
|
|
"search" : {
|
|
|
|
"request" : {
|
|
|
|
"indices" : [ "logs" ],
|
|
|
|
"body" : {
|
|
|
|
"query" : {
|
|
|
|
"match" : { "message": "error" }
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
},
|
|
|
|
"condition" : {
|
2018-12-05 13:49:06 -05:00
|
|
|
"compare" : { "ctx.payload.hits.total.value" : { "gt" : 0 }}
|
2017-03-28 17:23:01 -04:00
|
|
|
},
|
|
|
|
"actions" : {
|
|
|
|
"log_error" : {
|
|
|
|
"logging" : {
|
2018-12-05 13:49:06 -05:00
|
|
|
"text" : "Found {{ctx.payload.hits.total.value}} errors in the logs"
|
2017-03-28 17:23:01 -04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
[float]
|
|
|
|
[[log-delete]]
|
|
|
|
=== Delete the Watch
|
|
|
|
|
|
|
|
Since the `log_error_watch` is configured to run every 10 seconds, make sure you
|
|
|
|
delete it when you're done experimenting. Otherwise, the noise from this sample
|
|
|
|
watch will make it hard to see what else is going on in your watch history and
|
|
|
|
log file.
|
|
|
|
|
2019-09-30 13:18:50 -04:00
|
|
|
To remove the watch, use the <<watcher-api-delete-watch,delete watch API>>:
|
2017-03-28 17:23:01 -04:00
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2017-03-28 17:23:01 -04:00
|
|
|
--------------------------------------------------
|
2018-12-08 13:57:16 -05:00
|
|
|
DELETE _watcher/watch/log_error_watch
|
2017-03-28 17:23:01 -04:00
|
|
|
--------------------------------------------------
|
|
|
|
// TEST[continued]
|
|
|
|
|
2017-04-11 07:22:45 -04:00
|
|
|
[float]
|
|
|
|
[[required-security-privileges]]
|
2019-09-30 13:18:50 -04:00
|
|
|
=== Required security privileges
|
2017-06-27 20:16:51 -04:00
|
|
|
To enable users to create and manipulate watches, assign them the `watcher_admin`
|
2017-04-19 07:21:43 -04:00
|
|
|
security role. Watcher admins can also view watches, watch history, and triggered
|
|
|
|
watches.
|
2017-04-11 07:22:45 -04:00
|
|
|
|
2017-04-19 07:21:43 -04:00
|
|
|
To allow users to view watches and the watch history, assign them the `watcher_user`
|
|
|
|
security role. Watcher users cannot create or manipulate watches; they are only
|
|
|
|
allowed to execute read-only watch operations.
|
2017-04-11 07:22:45 -04:00
|
|
|
|
2017-03-28 17:23:01 -04:00
|
|
|
[float]
|
|
|
|
[[next-steps]]
|
2019-09-30 13:18:50 -04:00
|
|
|
=== Where to go next
|
2017-03-28 17:23:01 -04:00
|
|
|
|
2019-09-30 13:18:50 -04:00
|
|
|
* See <<how-watcher-works>> for more information about the
|
2017-03-28 17:23:01 -04:00
|
|
|
anatomy of a watch and the watch lifecycle.
|
2019-09-30 13:18:50 -04:00
|
|
|
* See <<example-watches>> for more examples of setting up
|
2017-03-28 17:23:01 -04:00
|
|
|
a watch.
|
|
|
|
* See the https://github.com/elastic/examples/tree/master/Alerting[Example
|
|
|
|
Watches] in the Elastic Examples repo for additional sample watches you can use
|
|
|
|
as a starting point for building custom watches.
|