18 KiB

Raw Blame History

layout	title	nav_order	parent	has_children
default	Monitors	1	Alerting	false

Monitors

TOC {:toc}

Key terms

Term	Definition
Monitor	A job that runs on a defined schedule and queries OpenSearch indices. The results of these queries are then used as input for one or more triggers.
Trigger	Conditions that, if met, generate alerts.
Alert	An event associated with a trigger. When an alert is created, the trigger performs actions, which can include sending a notification.
Action	The information that you want the monitor to send out after being triggered. Actions have a destination, a message subject, and a message body.
Channel	A notifications channel to use in an action. See notifications for more information.

Create monitors

Choose Alerting, Monitors, Create monitor.
Specify a name for the monitor.
Choose either Per query monitor or Per bucket monitor.

Whereas query-level monitors run your specified query and then check whether the query's results triggers any alerts, bucket-level monitors let you select fields to create buckets and categorize your results into those buckets. The alerting plugin runs each bucket's unique results against a script you define later, so you have finer control over which results should trigger alerts. Each of those buckets can trigger an alert, but query-level monitors can only trigger one alert at a time.

Define the monitor in one of three ways: visually, using a query, or using an anomaly detector.
- Visual definition works well for monitors that you can define as "some value is above or below some threshold for some amount of time."
- Query definition gives you flexibility in terms of what you query for (using the OpenSearch query DSL) and how you evaluate the results of that query (Painless scripting).
  
  This example averages the cpu_usage field:
```
{
  "size": 0,
  "query": {
    "match_all": {}
  },
  "aggs": {
    "avg_cpu": {
      "avg": {
        "field": "cpu_usage"
      }
    }
  }
}
```
  You can even filter query results using {% raw %}{{period_start}}{% endraw %} and {% raw %}{{period_end}}{% endraw %}:
```
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [{
        "range": {
          "timestamp": {
            "from": "{% raw %}{{period_end}}{% endraw %}||-1h",
            "to": "{% raw %}{{period_end}}{% endraw %}",
            "include_lower": true,
            "include_upper": true,
            "format": "epoch_millis",
            "boost": 1
          }
        }
      }],
      "adjust_pure_negative": true,
      "boost": 1
    }
  },
  "aggregations": {}
}
```
"Start" and "end" refer to the interval at which the monitor runs. See Available variables.

To define a monitor visually, choose Visual editor. Then choose a source index, a timeframe, an aggregation (for example, count() or average()), a data filter if you want to monitor a subset of your source index, and a group-by field if you want to include an aggregation field in your query. At least one group-by field is required if you're defining a bucket-level monitor. Visual definition works well for most monitors.

If you use the security plugin, you can only choose indices that you have permission to access. For details, see Alerting security.

To use a query, choose Extraction query editor, add your query (using the OpenSearch query DSL), and test it using the Run button.

The monitor makes this query to OpenSearch as often as the schedule dictates; check the Query Performance section and make sure you're comfortable with the performance implications.

To use an anomaly detector, choose Anomaly detector and select your Detector.

The anomaly detection option is for pairing with the anomaly detection plugin. See Anomaly Detection. For anomaly detector, choose an appropriate schedule for the monitor based on the detector interval. Otherwise, the alerting monitor might miss reading the results.

For example, assume you set the monitor interval and the detector interval as 5 minutes, and you start the detector at 12:00. If an anomaly is detected at 12:05, it might be available at 12:06 because of the delay between writing the anomaly and it being available for queries. The monitor reads the anomaly results between 12:00 and 12:05, so it does not get the anomaly results available at 12:06.

To avoid this issue, make sure the alerting monitor is at least twice the detector interval. When you create a monitor using OpenSearch Dashboards, the anomaly detector plugin generates a default monitor schedule that's twice the detector interval.

Whenever you update a detector’s interval, make sure to update the associated monitor interval as well, as the anomaly detection plugin does not do this automatically.

Note: Anomaly detection is available only if you are defining a per query monitor. {: .note}
Choose a frequency and timezone for your monitor. Note that you can only pick a timezone if you choose Daily, Weekly, Monthly, or custom cron expression for frequency.
Add a trigger to your monitor.

Create triggers

Steps to create a trigger differ depending on whether you chose Visual editor, Extraction query editor, or Anomaly detector when you created the monitor.

You begin by specifying a name and severity level for the trigger. Severity levels help you manage alerts. A trigger with a high severity level (e.g. 1) might page a specific individual, whereas a trigger with a low severity level might message a chat room.

Remember that query-level monitors run your trigger's script just once against the query's results, but bucket-level monitors execute your trigger's script on each bucket, so you should create a trigger that best fits the monitor you chose. If you want to execute multiple scripts, you must create multiple triggers.

Visual editor

For a query-level monitor's Trigger condition, specify a threshold for the aggregation and timeframe you chose earlier, such as "is below 1,000" or "is exactly 10."

The line moves up and down as you increase and decrease the threshold. Once this line is crossed, the trigger evaluates to true.

Bucket-level monitors also require you to specify a threshold and value for your aggregation and timeframe, but you can use a maximum of five conditions to better refine your trigger. Optionally, you can also use a keyword filter to filter for a specific field in your index.

Extraction query

If you're using a query-level monitor, specify a Painless script that returns true or false. Painless is the default OpenSearch scripting language and has a syntax similar to Groovy.

Trigger condition scripts revolve around the ctx.results[0] variable, which corresponds to the extraction query response. For example, your script might reference ctx.results[0].hits.total.value or ctx.results[0].hits.hits[i]._source.error_code.

A return value of true means the trigger condition has been met, and the trigger should execute its actions. Test your script using the Run button.

The Info link next to Trigger condition contains a useful summary of the variables and results available to your query. {: .tip }

Bucket-level monitors require you to specify more information in your trigger condition. At a minimum, you must have the following fields:

buckets_path, which maps variable names to metrics to use in your script.
parent_bucket_path, which is a path to a multi-bucket aggregation. The path can include single-bucket aggregations, but the last aggregation must be multi-bucket. For example, if you have a pipeline such as agg1>agg2>agg3, agg1 and agg2 are single-bucket aggregations, but agg3 must be a multi-bucket aggregation.
script, which is the script that OpenSearch runs to evaluate whether to trigger any alerts.

For example, you might have a script that looks like the following:

{
  "buckets_path": {
    "count_var": "_count"
  },
  "parent_bucket_path": "composite_agg",
  "script": {
    "source": "params.count_var > 5"
  }
}

After mapping the count_var variable to the _count metric, you can use count_var in your script and reference _count data. Finally, composite_agg is a path to a multi-bucket aggregation.

Anomaly detector

For Trigger type, choose Anomaly detector grade and confidence.

Specify the Anomaly grade condition for the aggregation and timeframe you chose earlier, "IS ABOVE 0.7" or "IS EXACTLY 0.5." The anomaly grade is a number between 0 and 1 that indicates the level of severity of how anomalous a data point is.

Specify the Anomaly confidence condition for the aggregation and timeframe you chose earlier, "IS ABOVE 0.7" or "IS EXACTLY 0.5." The anomaly confidence is an estimate of the probability that the reported anomaly grade matches the expected anomaly grade.

The line moves up and down as you increase and decrease the threshold. Once this line is crossed, the trigger evaluates to true.

Sample scripts

{::comment} These scripts are Painless, not Groovy, but calling them Groovy in Jekyll gets us syntax highlighting in the generated HTML. {:/comment}

// Evaluates to true if the query returned any documents
ctx.results[0].hits.total.value > 0

// Returns true if the avg_cpu aggregation exceeds 90
if (ctx.results[0].aggregations.avg_cpu.value > 90) {
  return true;
}

// Performs some crude custom scoring and returns true if that score exceeds a certain value
int score = 0;
for (int i = 0; i < ctx.results[0].hits.hits.length; i++) {
  // Weighs 500 errors 10 times as heavily as 503 errors
  if (ctx.results[0].hits.hits[i]._source.http_status_code == "500") {
    score += 10;
  } else if (ctx.results[0].hits.hits[i]._source.http_status_code == "503") {
    score += 1;
  }
}
if (score > 99) {
  return true;
} else {
  return false;
}

Below are some variables you can include in your message using Mustache templates to see more information about your monitors.

Available variables

Monitor variables

Variable	Data Type	Description
`ctx.monitor`	Object	Includes `ctx.monitor.name`, `ctx.monitor.type`, `ctx.monitor.enabled`, `ctx.monitor.enabled_time`, `ctx.monitor.schedule`, `ctx.monitor.inputs`, `triggers` and `ctx.monitor.last_update_time`.
`ctx.monitor.user`	Object	Includes information about the user who created the monitor. Includes `ctx.monitor.user.backend_roles` and `ctx.monitor.user.roles`, which are arrays that contain the backend roles and roles assigned to the user. See alerting security for more information.
`ctx.monitor.enabled`	Boolean	Whether the monitor is enabled.
`ctx.monitor.enabled_time`	Milliseconds	Unix epoch time of when the monitor was last enabled.
`ctx.monitor.schedule`	Object	Contains a schedule of how often or when the monitor should run.
`ctx.monitor.schedule.period.interval`	Integer	The interval at which the monitor runs.
`ctx.monitor.schedule.period.unit`	String	The interval's unit of time.
`ctx.monitor.inputs`	Array	An array that contains the indices and definition used to create the monitor.
`ctx.monitor.inputs.search.indices`	Array	An array that contains the indices the monitor observes.
`ctx.monitor.inputs.search.query`	N/A	The definition used to define the monitor.

Trigger variables

Action variables

Other variables

Variable | Data Type | Description :--- | :--- : :--- ctx.results | Array | An array with one element (i.e. ctx.results[0]). Contains the query results. This variable is empty if the trigger was unable to retrieve results. See ctx.error. ctx.last_update_time | Milliseconds | Unix epoch time of when the monitor was last updated. ctx.periodStart | String | Unix timestamp for the beginning of the period during which the alert triggered. For example, if a monitor runs every ten minutes, a period might begin at 10:40 and end at 10:50. ctx.periodEnd | String | The end of the period during which the alert triggered. ctx.error | String | The error message if the trigger was unable to retrieve results or unable to evaluate the trigger, typically due to a compile error or null pointer exception. Null otherwise. ctx.alert | Object | The current, active alert (if it exists). Includes ctx.alert.id, ctx.alert.version, and ctx.alert.isAcknowledged. Null if no alert is active. Only available with query-level monitors. ctx.dedupedAlerts | Object | Alerts that have already been triggered. OpenSearch keeps the existing alert to prevent the plugin from creating endless amounts of the same alerts. Only available with bucket-level monitors. ctx.newAlerts | Object | Newly created alerts. Only available with bucket-level monitors. ctx.completedAlerts | Object | Alerts that are no longer ongoing. Only available with bucket-level monitors. bucket_keys | String | Comma-separated list of the monitor's bucket key values. Available only for ctx.dedupedAlerts, ctx.newAlerts, and ctx.completedAlerts. Accessed through ctx.dedupedAlerts[0].bucket_keys. parent_bucket_path | String | The parent bucket path of the bucket that triggered the alert. Accessed through ctx.dedupedAlerts[0].parent_bucket_path.

Add actions

The final step in creating a monitor is to add one or more actions. Actions send notifications when trigger conditions are met and support Slack, Amazon Chime, and webhooks.

If you don't want to receive notifications for alerts, you don't have to add actions to your triggers. Instead, you can periodically check OpenSearch Dashboards. {: .tip }

Specify a name for the action.
Choose a notification channel.
Add a subject and body for the message.

You can add variables to your messages using Mustache templates. You have access to ctx.action.name, the name of the current action, as well as all trigger variables.

If your destination is a custom webhook that expects a particular data format, you might need to include JSON (or even XML) directly in the message body:
```
{% raw %}{ "text": "Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue. - Trigger: {{ctx.trigger.name}} - Severity: {{ctx.trigger.severity}} - Period start: {{ctx.periodStart}} - Period end: {{ctx.periodEnd}}" }{% endraw %}
```
In this case, the message content must conform to the Content-Type header in the custom webhook.
If you're using a bucket-level monitor, you can choose whether the monitor should perform an action for each execution or for each alert.
(Optional) Use action throttling to limit the number of notifications you receive within a given span of time.

For example, if a monitor checks a trigger condition every minute, you could receive one notification per minute. If you set action throttling to 60 minutes, you receive no more than one notification per hour, even if the trigger condition is met dozens of times in that hour.
Choose Create.

After an action sends a message, the content of that message has left the purview of the security plugin. Securing access to the message (e.g. access to the Slack channel) is your responsibility.

Sample message

{% raw %}Monitor {{ctx.monitor.name}} just entered an alert state. Please investigate the issue.
- Trigger: {{ctx.trigger.name}}
- Severity: {{ctx.trigger.severity}}
- Period start: {{ctx.periodStart}}
- Period end: {{ctx.periodEnd}}{% endraw %}

If you want to use the ctx.results variable in a message, use {% raw %}{{ctx.results.0}}{% endraw %} rather than {% raw %}{{ctx.results[0]}}{% endraw %}. This difference is due to how Mustache handles bracket notation. {: .note }

Work with alerts

Alerts persist until you resolve the root cause and have the following states:

State	Description
Active	The alert is ongoing and unacknowledged. Alerts remain in this state until you acknowledge them, delete the trigger associated with the alert, or delete the monitor entirely.
Acknowledged	Someone has acknowledged the alert, but not fixed the root cause.
Completed	The alert is no longer ongoing. Alerts enter this state after the corresponding trigger evaluates to false.
Error	An error occurred while executing the trigger---usually the result of a a bad trigger or destination.
Deleted	Someone deleted the monitor or trigger associated with this alert while the alert was ongoing.

18 KiB Raw Blame History Unescape Escape