230 lines
6.8 KiB
Plaintext
230 lines
6.8 KiB
Plaintext
[[watching-time-series-data]]
|
|
=== Watching Time Series Data
|
|
|
|
If you are indexing time-series data such as logs, RSS feeds, or network traffic,
|
|
you can use {watcher} to send notifications when certain events occur.
|
|
|
|
For example, you could index an RSS feed of posts on Stack Overflow that are
|
|
tagged with Elasticsearch, Logstash, Beats, or Kibana, set up a watch to check
|
|
daily for new posts about a problem or failure, and send an email if any are
|
|
found.
|
|
|
|
The simplest way to index an RSS feed is to use https://www.elastic.co/products/logstash[Logstash].
|
|
|
|
To install Logstash and set up the RSS input plugin:
|
|
|
|
. https://www.elastic.co/downloads/logstash[Download Logstash] and unpack the
|
|
archive file.
|
|
. Go to the `logstash-{version}` directory and install the
|
|
{logstash-ref}/plugins-inputs-rss.html[RSS input] plugin:
|
|
+
|
|
[source,sh]
|
|
----------------------------------------------------------
|
|
cd logstash-<logstash_version>
|
|
bin/logstash-plugin install logstash-input-rss
|
|
----------------------------------------------------------
|
|
|
|
. Create a Logstash configuration file that uses the RSS input plugin to get
|
|
data from an RSS/atom feed and outputs the data to Elasticsearch. For example,
|
|
the following `rss.conf` file gets events from the Stack Overflow feed that
|
|
are tagged with `elasticsearch`, `logstash`, `beats` or `kibana`.
|
|
+
|
|
[source,ruby]
|
|
----------------------------------------------------------
|
|
input {
|
|
rss {
|
|
url => "http://stackoverflow.com/feeds/tag/elasticsearch+or+logstash+or+beats+or+kibana"
|
|
interval => 3600 <1>
|
|
}
|
|
}
|
|
|
|
output {
|
|
elasticsearch { }
|
|
stdout { }
|
|
}
|
|
----------------------------------------------------------
|
|
<1> Checks the feed every hour.
|
|
+
|
|
For more information see {logstash-ref}/plugins-outputs-elasticsearch.html[Elasticsearch output]
|
|
in the Logstash Reference.
|
|
|
|
. Run Logstash with the `rss.conf` config file to start indexing the feed:
|
|
+
|
|
[source,she]
|
|
----------------------------------------------------------
|
|
bin/logstash -f rss.conf
|
|
----------------------------------------------------------
|
|
|
|
Once you have Logstash set up to input data from the RSS feed into Elasticsearch,
|
|
you can set up a daily watch that runs at noon to check for new posts that
|
|
contain the words "error" or "problem".
|
|
|
|
To set up the watch:
|
|
|
|
. Define the watch trigger--a daily schedule that runs at 12:00 UTC:
|
|
+
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"trigger" : {
|
|
"schedule" : {
|
|
"daily" : { "at" : "12:00" }
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
+
|
|
NOTE: In {watcher}, you specify times in UTC time. Don't forget to do the
|
|
conversion from your local time so the schedule triggers at the time
|
|
you intend.
|
|
|
|
. Define the watch input--a search that uses a filter to constrain the results
|
|
to the past day.
|
|
+
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"input" : {
|
|
"search" : {
|
|
"request" : {
|
|
"indices" : [ "logstash*" ],
|
|
"body" : {
|
|
"query" : {
|
|
"bool" : {
|
|
"must" : { "match" : { "message": "error problem" }},
|
|
"filter" : { "range" : { "@timestamp" : { "gte" : "now-1d" }}}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
. Define a watch condition to check the payload to see if the input search
|
|
returned any hits. If it did, the condition resolves to `true` and the watch
|
|
actions will be executed.
|
|
+
|
|
You define the condition with the following script:
|
|
+
|
|
[source,text]
|
|
--------------------------------------------------
|
|
return ctx.payload.hits.total > threshold
|
|
--------------------------------------------------
|
|
+
|
|
If you store the script in a file at `$ES_HOME/config/scripts/threshold_hits.painless`,
|
|
you can then reference it by name in the watch condition.
|
|
+
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"condition" : {
|
|
"script" : {
|
|
"id" : "threshold_hits",
|
|
"params" : {
|
|
"threshold" : 0 <1>
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
<1> The threshold parameter value you want to pass to the script.
|
|
+
|
|
. Define a watch action to send an email that contains the relevant messages
|
|
from the past day as an attachment.
|
|
+
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"actions" : {
|
|
"send_email" : {
|
|
"email" : {
|
|
"to" : "<username>@<domainname>",
|
|
"subject" : "Somebody needs help with the Elastic Stack",
|
|
"body" : "The attached Stack Overflow posts were tagged with Elasticsearch, Logstash, Beats or Kibana and mentioned an error or problem.",
|
|
"attachments" : {
|
|
"attached_data" : {
|
|
"data" : {
|
|
"format" : "json"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
+
|
|
NOTE: To use the email action, you must configure at least one email account in
|
|
`elasticsearch.yml`. If you configure multiple email accounts, you need to
|
|
specify which one you want to send the email with. For more information, see
|
|
<<configuring-email, Working with Various Email Services>>.
|
|
|
|
The complete watch looks like this:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _xpack/watcher/watch/rss_watch
|
|
{
|
|
"trigger" : {
|
|
"schedule" : {
|
|
"daily" : { "at" : "12:00" }
|
|
}
|
|
},
|
|
"input" : {
|
|
"search" : {
|
|
"request" : {
|
|
"indices" : [ "logstash*" ],
|
|
"body" : {
|
|
"query" : {
|
|
"bool" : {
|
|
"must" : { "match" : { "message": "error problem" }},
|
|
"filter" : { "range" : { "@timestamp" : { "gte" : "now-1d" }}}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"condition" : {
|
|
"script" : {
|
|
"id" : "threshold_hits",
|
|
"params" : {
|
|
"threshold" : 0
|
|
}
|
|
}
|
|
},
|
|
"actions" : {
|
|
"send_email" : {
|
|
"email" : {
|
|
"to" : "<username>@<domainname>", <1>
|
|
"subject" : "Somebody needs help with the Elastic Stack",
|
|
"body" : "The attached Stack Overflow posts were tagged with Elasticsearch, Logstash, Beats or Kibana and mentioned an error or problem.",
|
|
"attachments" : {
|
|
"attached_data" : {
|
|
"data" : {}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[s/"id" : "threshold_hits"/"source": "return ctx.payload.hits.total > params.threshold"/]
|
|
<1> Replace `<username>@<domainname>` with your email address to receive
|
|
notifications.
|
|
|
|
[TIP]
|
|
=================================================
|
|
To execute a watch immediately (without waiting for the schedule to trigger),
|
|
use the {ref}/watcher-api-execute-watch.html[`_execute` API]:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
POST _xpack/watcher/watch/rss_watch/_execute
|
|
{
|
|
"ignore_condition" : true,
|
|
"action_modes" : {
|
|
"_all" : "force_execute"
|
|
},
|
|
"record_execution" : true
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
=================================================
|