287 lines
9.4 KiB
Plaintext
287 lines
9.4 KiB
Plaintext
[[watching-meetup-data]]
|
|
=== Watching Event Data
|
|
|
|
If you are indexing event data, such as log messages, network traffic, or a web feed, you can create a watch to email notifications when certain events occur.
|
|
For example, if you index a feed of RSVPs for meetup events happening around the world, you can create a watch that alerts you to interesting events.
|
|
|
|
To index the meetup data, you can use https://www.elastic.co/products/logstash[Logstash] to ingest live data from the Meetup.com streaming API, `http://stream.meetup.com/2/rsvps`.
|
|
|
|
To ingest this data with Logstash:
|
|
|
|
. https://www.elastic.co/downloads/logstash[Download Logstash] and unpack the
|
|
archive file.
|
|
|
|
. Create a Logstash configuration file that uses the {logstash-ref}/plugins-inputs-stdin.html[Logstash standard input] and the {logstash-ref}/plugins-outputs-stdout.html[Logstash standard output] and save it in `logstash-{version}` directory as `livestream.conf`:
|
|
+
|
|
[source,ruby]
|
|
----------------------------------------------------------
|
|
input {
|
|
stdin {
|
|
codec => json <1>
|
|
}
|
|
}
|
|
filter {
|
|
date {
|
|
match => [ "event.time", "UNIX_MS" ]
|
|
target => "event_time"
|
|
}
|
|
}
|
|
output { <2>
|
|
stdout {
|
|
codec => rubydebug
|
|
}
|
|
elasticsearch {
|
|
hosts => "http://localhost:9200"
|
|
user => "elastic"
|
|
password => "x-pack-test-password"
|
|
}
|
|
}
|
|
|
|
----------------------------------------------------------
|
|
<1> The meetup data stream is formatted in JSON.
|
|
<2> Index the meetup data into Elasticsearch.
|
|
|
|
. To start indexing the meetup data, pipe the RSVP stream into Logstash and specify your `livestream.conf` configuration file.
|
|
+
|
|
[source,she]
|
|
----------------------------------------------------------
|
|
|
|
curl http://stream.meetup.com/2/rsvps | bin/logstash -f livestream.conf
|
|
---------------------------------------------------------
|
|
|
|
Now that you're indexing the meetup RSVPs, you can set up a watch that lets you know about events you might be interested in. For example, let's create a watch that runs every hour, looks for events that talk about about _Open Source_, and sends an email with information about the events.
|
|
|
|
|
|
To set up the watch:
|
|
|
|
. Specify how often you want to run the watch by adding a schedule trigger to the watch:
|
|
+
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"trigger": {
|
|
"schedule": {
|
|
"interval": "1h"
|
|
}
|
|
},
|
|
--------------------------------------------------
|
|
. Load data into the watch payload by creating an input that searches the meetup data for events that have _Open Source_ as a topic. You can use aggregations to group the data by city, consolidate references to the same events, and sort the events by date.
|
|
+
|
|
[source,js]
|
|
-------------------------------------------------
|
|
"input": {
|
|
"search": {
|
|
"request": {
|
|
"indices": [
|
|
"<logstash-{now-1h}>", <1>
|
|
"<logstash-{now}>"
|
|
],
|
|
"body": {
|
|
"size": 0,
|
|
"query": {
|
|
"bool": {
|
|
"filter": [
|
|
{
|
|
"range": {
|
|
"@timestamp": {
|
|
"gte": "now-3h"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"match": {
|
|
"group.group_topics.topic_name": "Open Source" <2>
|
|
}
|
|
}
|
|
]
|
|
}
|
|
},
|
|
"aggs": {
|
|
"group_by_city": {
|
|
"terms": {
|
|
"field": "group.group_city.raw", <3>
|
|
"size": 5
|
|
},
|
|
"aggs": {
|
|
"group_by_event": {
|
|
"terms": {
|
|
"field": "event.event_url.raw", <4>
|
|
"size": 5
|
|
},
|
|
"aggs": {
|
|
"get_latest": {
|
|
"terms": {
|
|
"field": "@timestamp", <5>
|
|
"size": 1,
|
|
"order": {
|
|
"_key": "desc"
|
|
}
|
|
},
|
|
"aggs": {
|
|
"group_by_event_name": {
|
|
"terms": {
|
|
"field": "event.event_name.raw" <6>
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
},
|
|
--------------------------------------------------
|
|
<1> Elasticsearch Date math is used to select the Logstash indices that contain the meetup data. The second pattern is needed in case the previous hour crosses days.
|
|
<2> Find all of the RSVPs with `Open Source` as a topic.
|
|
<3> Group the RSVPs by city.
|
|
<4> Consolidate multiple RSVPs for the same event.
|
|
<5> Sort the events so the latest events are listed first.
|
|
<6> Group the events by name.
|
|
|
|
. To determine whether or not there are any Open Source events, add a compare condition that checks the watch payload to see if there were any search hits.
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
|
|
--------------------------------------------------
|
|
. To send an email when _Open Source_ events are found, add an email action:
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"actions": {
|
|
"email_me": {
|
|
"throttle_period": "10m",
|
|
"email": {
|
|
"from": "<from:email address>",
|
|
"to": "<to:email address>",
|
|
"subject": "Open Source Events",
|
|
"body": {
|
|
"html": "Found events matching Open Source: <ul>{{#ctx.payload.aggregations.group_by_city.buckets}}< li>{{key}} ({{doc_count}})<ul>{{#group_by_event.buckets}}
|
|
<li><a href=\"{{key}}\">{{get_latest.buckets.0.group_by_event_name.buckets.0.key}}</a>
|
|
({{doc_count}})</li>{{/group_by_event.buckets}}</ul></li>
|
|
{{/ctx.payload.aggregations.group_by_city.buckets}}</ul>"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
---------------------------------------------------
|
|
|
|
NOTE: To enable Watcher to send emails, you must configure an email account in `elasticsearch.yml`. For more information, see <<configuring-email, Working with Various Email Services>>.
|
|
|
|
|
|
The complete watch looks like this:
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _xpack/watcher/watch/meetup
|
|
{
|
|
"trigger": {
|
|
"schedule": {
|
|
"interval": "1h"
|
|
}
|
|
},
|
|
"input": {
|
|
"search": {
|
|
"request": {
|
|
"indices": [
|
|
"<logstash-{now-1h}>",
|
|
"<logstash-{now}>"
|
|
],
|
|
"body": {
|
|
"size": 0,
|
|
"query": {
|
|
"bool": {
|
|
"filter": [
|
|
{
|
|
"range": {
|
|
"@timestamp": {
|
|
"gte": "now-3h"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"match": {
|
|
"group.group_topics.topic_name": "Open Source"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
},
|
|
"aggs": {
|
|
"group_by_city": {
|
|
"terms": {
|
|
"field": "group.group_city.raw",
|
|
"size": 5
|
|
},
|
|
"aggs": {
|
|
"group_by_event": {
|
|
"terms": {
|
|
"field": "event.event_url.raw",
|
|
"size": 5
|
|
},
|
|
"aggs": {
|
|
"get_latest": {
|
|
"terms": {
|
|
"field": "@timestamp",
|
|
"size": 1,
|
|
"order": {
|
|
"_key": "desc"
|
|
}
|
|
},
|
|
"aggs": {
|
|
"group_by_event_name": {
|
|
"terms": {
|
|
"field": "event.event_name.raw"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"condition": {
|
|
"compare": {
|
|
"ctx.payload.hits.total": {
|
|
"gt": 0
|
|
}
|
|
}
|
|
},
|
|
"actions": { <1>
|
|
"email_me": {
|
|
"throttle_period": "10m",
|
|
"email": {
|
|
"from": "<username>@<domainname>", <2>
|
|
"to": "<username@<domainname>", <3>
|
|
"subject": "Open Source events",
|
|
"body": {
|
|
"html": "Found events matching Open Source: <ul>{{#ctx.payload.aggregations.group_by_city.buckets}}<li>{{key}} ({{doc_count}})<ul>{{#group_by_event.buckets}}<li><a href=\"{{key}}\">{{get_latest.buckets.0.group_by_event_name.buckets.0.key}}</a> ({{doc_count}})</li>{{/group_by_event.buckets}}</ul></li>{{/ctx.payload.aggregations.group_by_city.buckets}}</ul>"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
<1> The email body can include Mustache templates to reference data in the watch payload. By default,it will be <<email-html-sanitization, sanitized>> to block dangerous content.
|
|
<2> Replace the `from` address with the email address you configured in `elasticsearch.yml`.
|
|
<3> Replace the `to` address with your email address to receive notifications.
|
|
|
|
|
|
Now that you've created your watch, you can use the
|
|
{ref}/watcher-api-execute-watch.html[`_execute` API] to run it without waiting for the schedule to trigger execution:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
POST _xpack/watcher/watch/meetup/_execute
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|