OpenSearch/watcher/docs/example-watches/watching-time-series-data.asciidoc

[[watching-time-series-data]]
=== Watching Time Series Data

If you are indexing time-series data such as logs, RSS feeds, or network traffic,
you can use watcher to send notifications when certain events occur. 

For example, you could index an RSS feed of posts on Stack Overflow that are tagged with Elasticsearch, Logstash, or Kibana, set up a watch to check daily for new posts about a problem or failure, and send an email if any are found.

The simplest way to index an RSS feed is to use https://www.elastic.co/products/logstash[Logstash]. 

To install Logstash and set up the RSS input plugin:

. https://www.elastic.co/downloads/logstash[Download Logstash 1.5.0 RC4+] and unpack the  archive file.
. Go to the `logstash-<logstash_version>` directory and install the 
http://www.elastic.co/guide/en/logstash/current/plugins-inputs-rss.html[RSS input]
plugin:
+
[source,shell]
----------------------------------------------------------
cd logstash-<logstash_version>
bin/plugin install logstash-input-rss
----------------------------------------------------------

. Create a Logstash configuration file that uses the RSS input plugin
to get data from an RSS/atom feed and outputs the data to Elasticsearch. For example, the following `rss.conf` file gets events from the Stack Overflow feed that are tagged with `elasticsearch`, `logstash`, or `kibana`.
+
[source,text]
----------------------------------------------------------
input {
  rss {
    url => "http://stackoverflow.com/feeds/tag/elasticsearch+or+logstash+or+kibana"
    interval => 3600 <1>
  }
}

output {
  elasticsearch { }
  stdout { }
}
----------------------------------------------------------
<1> Checks the feed every hour. 
+
For more information see {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-host[Elasticsearch output] in the Logstash Reference.

. Run Logstash with the `rss.conf` config file to start indexing the feed:
+
[source,shell]
----------------------------------------------------------
bin/logstash -f rss.conf
----------------------------------------------------------

Once you have Logstash set up to input data from the RSS feed into 
Elasticsearch, you can set up a watch that runs at noon each day to check for new posts that contain the words "error" or "problem".

To set up the watch, define a trigger, input, condition, and an action: 

. Define the watch trigger--a daily schedule that runs at 12:00 UTC time every day:
+
[source,json]
--------------------------------------------------
"trigger" : {    
  "schedule" : {
    "daily" : { "at" : "12:00" }
  }
}
--------------------------------------------------
+
NOTE: In Watcher, you specify times in UTC time. Don't forget to do the conversion from your local time so the schedule triggers at the time you intend.

. Define the watch input--a search that uses a filter to constrain the results to
the past day.
+
[source,json]
--------------------------------------------------
"input" : {
  "search" : {
    "request" : {
      "indices" : [ "logstash*" ],
      "body" : {
        "query" : {
          "filtered" : {
            "query" : {"match" : {"message": "error problem"}},
            "filter" : {
              "range" : {"@timestamp" : {"gte" : "now-1d"}}
            }
          }
        }
      }
    }
  }
}
--------------------------------------------------

. Define a watch condition to check the payload to see if the input search returned any hits. If it did, the condition resolves to `true` and the watch actions will be performed. 
+
You define the condition with the following script:
+
[source,text]
--------------------------------------------------
return ctx.payload.hits.total > threshold
--------------------------------------------------
+
If you store the script in a file at `$ES_HOME/config/scripts/threshold_hits.groovy`, you can then reference it by name in the watch condition. Using file-based Groovy scripts enables you to avoid using dynamic scripting. For more information see {blog-ref}running-groovy-scripts-without-dynamic-scripting[Running Groovy Scripts without Dynamic Scripting].
+
[source,json]
--------------------------------------------------
"condition" : {
    "script" : {
      "file" : "threshold_hits",
      "params" : {
        "threshold" : 0 <1>
      }
    }
  },
--------------------------------------------------
+
<1> The threshold parameter value you want to pass to the script.
+
NOTE: We recommend using file scripts when possible. To use inline or indexed scripts, you must {ref}/modules-scripting.html[enable dynamic scripting] in Elasticsearch.

. Define a watch action to send an email that contains the relevant messages from the past day as an attachment. 
+
[source,json]
--------------------------------------------------
"actions" : {
  "send_email" : {
    "email" : {
      "to" : "<username>@<domainname>",
      "subject" : "Somebody needs help with ELK", 
      "body" : "The attached Stack Overflow posts were tagged with Elasticsearch, Logstash, or Kibana and mentioned an error or problem.",
      "attach_data" : true
    } 
  } 
} 
--------------------------------------------------
+
NOTE: To use the email action, you must configure at least one email account in 
`elasticsearch.yml`. If you configure multiple email accounts, you need to specify which one you want to use in the email action. For more information, see <<email-services, Working with Various Email Services>>.

The complete watch looks like this:

[source,json]
--------------------------------------------------
PUT _watcher/watch/rss_watch
{
  "trigger" : {
      "schedule" : {
        "daily" : { "at" : "12:00" }
    }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "logstash*" ],
        "body" : {
          "query" : {
            "filtered" : {
              "query" : {"match" : {"message": "error problem"}},
              "filter" : {"range" : {"@timestamp" : {"gte" : "now-1d"}}}
            }
          }
        }
      }
    }
  },
  "condition" : {
    "script" : {
      "file" : "threshold_hits",
      "params" : {
        "threshold" : 0
      }
    }
  },
  "actions" : {
    "send_email" : {
      "email" : {
        "to" : "<username>@<domainname>",  <1>
        "subject" : "Somebody needs help with ELK", 
        "body" : "The attached Stack Overflow posts were tagged with Elasticsearch, Logstash, or Kibana and mentioned an error or problem.",
        "attach_data" : true
      }
    }
  }
}
--------------------------------------------------
// AUTOSENSE

<1> Replace `<username>@<domainname>` with your email address to receive notifications.

[TIP]
=================================================
To execute a watch immediately (without waiting for the schedule to trigger), use the <<api-rest-execute-watch, `_execute`>> API:

[source,json]
--------------------------------------------------
POST _watcher/watch/rss_watch/_execute
{
  "ignore_condition" : true,
  "action_modes" : {
    "_all" : "force_execute"
  },
  "record_execution" : true
}
--------------------------------------------------
// AUTOSENSE
==================================================