iSharkFly-Docs
/
opensearch-docs-cn
mirror of https://github.com/iSharkFly-Docs/opensearch-docs-cn
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #604 from alicejw-aws/475-doc-level-alerts 

doc-level monitor API - findings index and search API
This commit is contained in:
Alice Williams 2022-05-26 13:59:05 -07:00 committed by GitHub
parent 035dfa8ee1 95a614ebb5
commit c80ab9739a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 200 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

127
_monitoring-plugins/alerting/api.md
View File

@ -7,7 +7,7 @@ nav_order: 15
# Alerting API
Use the alerting API to programmatically manage monitors and alerts.
Use the Alerting API to programmatically create, update, and manage monitors and alerts.
---
@ -18,13 +18,13 @@ Use the alerting API to programmatically manage monitors and alerts.
---
## Create query-level monitor
## Create a query-level monitor
Introduced 1.0
{: .label .label-purple }
Query-level monitors run the query and check whether the results should trigger any alerts. As such, query-level monitors can only trigger one alert at a time. For more information about query-level monitors versus bucket-level monitors, see [Create monitors]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/#create-monitors).
Query-level monitors run the query and check whether or not the results should trigger an alert. Query-level monitors can only trigger one alert at a time. For more information about query-level monitors and bucket-level monitors, see [Create monitors]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/#create-monitors).
#### Request
#### Sample Request
```json
POST _plugins/_alerting/monitors
@ -250,15 +250,15 @@ For a full list of timezone names, refer to [Wikipedia](https://en.wikipedia.org
---
## Create bucket-level monitor
## Create a bucket-level monitor
Bucket-level monitors categorize results into buckets separated by fields. The monitor then runs your script with each bucket's results and evaluates whether to trigger an alert. For more information about bucket-level monitors versus query-level monitors, see [Create monitors]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/#create-monitors).
Bucket-level monitors categorize results into buckets separated by fields. The monitor then runs your script with each bucket's results and evaluates whether to trigger an alert. For more information about bucket-level and query-level monitors, see [Create monitors]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/#create-monitors).
```json
POST _plugins/_alerting/monitors
{
"type": "monitor",
"name": "test-bucket-level-monitor",
"name": "Demo bucket-level monitor",
"monitor_type": "bucket_level_monitor",
"enabled": true,
"schedule": {
@ -379,7 +379,7 @@ POST _plugins/_alerting/monitors
"monitor" : {
"type" : "monitor",
"schema_version" : 4,
"name" : "test-bucket-level-monitor",
"name" : "Demo a bucket-level monitor",
"monitor_type" : "bucket_level_monitor",
"user" : {
"name" : "",
@ -508,28 +508,70 @@ POST _plugins/_alerting/monitors
}
}
```
## Create a document-level monitor
## Document-level monitors
Introduced 2.0
{: .label .label-purple }
You can create a document-level monitor programmatically that functions the same as a per document monitor in OpenSearch Dashboards. The document-level monitor runs a query that checks whether or not the results should trigger an alert notification based on individual documents within an index.
Document-level monitors check whether individual documents in an index match trigger conditions. If so, the monitor generates an alert notification. When you run a query with a document-level monitor, the results are returned for each document that matches the trigger condition. You can create trigger conditions based on query names, query IDs, or tags that combine multiple queries.
You can combine multiple queries in a per document monitor by adding a tag to each query and then setting the trigger condition to that tag.
To learn more about per document monitors that function similarly to the document-level monitor API, see [Monitor types]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/#monitor-types).
For more information about document-level monitors, see [Monitor types]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/#monitor-types).
### Search for monitor findings
#### Sample request
You can use the Alerting search API operation to search the findings index `.opensearch-alerting-finding*` for available document findings with a GET request. By default, a GET request without path parameters returns all available findings. To learn more about monitor findings, see [Document findings]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/#document-findings).
The following sample shows how to create a per document monitor:
To retrieve any available findings, send a GET request without any path parameters as follows:
```json
GET /_plugins/_alerting/findings/_search?
```
To retrieve metadata for an individual document finding entry, you can search for the finding by its `findingId` as follows:
```json
GET /_plugins/_alerting/findings/_search?findingId=gKQhj8WJit3BxjGfiOXC
```
The response returns the number of individual finding entries in the `total_findings` field.
To get more specific results in a findings search, you can use any of the optional path parameters that are defined in the following table:
Path parameter | Description | Usage
:--- | :--- : :---
`findingId` | The identifier for the finding entry. | The finding ID is returned in the initial query response.
`sortString` | This field specifies which string the Alerting plugin uses to sort the findings. | The default value is `id`.
`sortOrder` | The order to sort the list of findings, either ascending or descending. | Use `sortOrder=asc` to indicate ascending, or `sortOrder=desc` for descending sort order.
`size` | An optional limit for the maximum number of results returned in the response. | There is no minimum or maximum values.
`startIndex` | The pagination indicator. | Default is `0`.
`searchString` | The finding attribute you want returned in the search. | To search in a specific index, specify the index name in the request path. For example, to search findings in the `indexABC` index, use `searchString=indexABC'.
### Create a document-level monitor
You can create a document-level monitor with a POST request that provides the monitor details in the request body.
At a minimum, you need to provide the following details: specify the queries or combinations by tag with the `inputs` field, a valid trigger condition, and provide the notification message in the `action` field.
The following table shows the syntax to use for each trigger option:
Trigger options | Definition | Syntax
:--- | :--- : :---
Tag | Creates alerts for documents that match a multiple query with this tag applied. If you group multiple queries by a single tag, then you can set it to trigger an alert if the results are returned by this tag name.| `query[tag=<tag-name>]`
Query by name | Creates alerts for documents matched or returned by the named query. | `query[name=<query-name>]`
Query by ID | Creates alerts for documents that were returned by the identified query. | `query[id=<query-id>]`
#### Sample Request
The following sample shows how to create a document-level monitor:
```json
POST _plugins/_alerting/monitors
{
"type": "monitor",
"monitor_type": "doc_level_monitor",
"name": "iad-monitor",
"name": "Example document-level monitor",
"enabled": true,
"createdBy": "chip",
"schedule": {
"period": {
"interval": 1,
@ -539,24 +581,34 @@ POST _plugins/_alerting/monitors
"inputs": [
{
"doc_level_input": {
"description": "windows-powershell",
"description": "Example document-level monitor for audit logs",
"indices": [
"test-logs"
"audit-logs"
],
"queries": [
{
"id": "sigma-123",
"id": "nKQnFYABit3BxjGfiOXC",
"name": "sigma-123",
"query": "region:\"us-west-2\"",
"tags": [
"MITRE:8500"
],
"tag1"
]
},
{
"id": "sigma-456",
"id": "gKQnABEJit3BxjGfiOXC",
"name": "sigma-456",
"query": "region:\"us-east-1\"",
"tags": [
"MITRE:8600"
],
"tag2"
]
},
{
"id": "h4J2ABEFNW3vxjGfiOXC",
"name": "sigma-789",
"query": "message:\"This is a SEPARATE error from IAD region\"",
"tags": [
"tag3"
]
}
]
}
@ -567,18 +619,39 @@ POST _plugins/_alerting/monitors
"severity": "1",
"condition": {
"script": {
"source": "'sigma-123' && !'sigma-456'",
"source": "(query[name=sigma-123] || query[tag=tag3]) && query[name=sigma-789]",
"lang": "painless"
}
},
"actions": []
"actions": [
{
"name": "test-action",
"destination_id": "E4o5hnsB6KjPKmHtpfCA",
"message_template": {
"source": """Monitor just entered alert status. Please investigate the issue. Related Finding Ids: {{ctx.alerts.0.finding_ids}}, Related Document Ids: {{ctx.alerts.0.related_doc_ids}}""",
"lang": "mustache"
},
"action_execution_policy": {
"action_execution_scope": {
"per_alert": {
"actionable_alerts": []
}
}
},
"subject_template": {
"source": "The Subject",
"lang": "mustache"
}
}
]
}}]
}
```
### Limitations
If you run a document-level query while the index is getting reindexed, the results will not return the reindexed results. You would need to run the query again to get the updates.
If you run a document-level query while the index is getting reindexed, the API response will not return the reindexed results. To get updates, wait until the reindexing process completes, then rerun the query.
{: .tip}
## Update monitor

107
_monitoring-plugins/alerting/monitors.md
View File

@ -22,18 +22,6 @@ The OpenSearch Dashboard Alerting plugin provides four monitor types:
* **per cluster metrics** – This monitor runs API requests on the cluster to monitor its health.
* **per document** – This monitor runs a query (or multiple queries combined by a tag) that returns individual documents that match the alert notification trigger condition.
### More about per document monitors
The per query and per bucket monitors can only take a single query with one trigger condition. Per document monitors allow you to combine multiple query trigger conditions by adding a tag to the queries. Then you can add the tag as a single trigger condition instead of specifying a single query. The Alerting plugin processes the trigger conditions from all queries as a logical OR operation, so if any of the query conditions are met, it generates the alert notification.
The Alerting plugin also creates a list of document findings data that contains metadata about which document matches each query. Security analytics can use the document findings data to keep track of and analyze the query data separately from the alert processes.
The following metadata is provided for each document finding entry:
* **Document** – The document ID and index name.
* **Query** – The query name that matched the document.
* **Time found** – The timestamp that indicates when the document was found during the runtime.
## Key terms
Term | Definition
@ -43,8 +31,103 @@ Trigger | Conditions that, if met, generate *alerts*.
Tag | A label that can be applied to multiple queries to combine them with the logical OR operation in a per document monitor. You cannot use tags with other monitor types.
Alert | An event associated with a trigger. When an alert is created, the trigger performs *actions*, which can include sending a notification.
Action | The information that you want the monitor to send out after being triggered. Actions have a *destination*, a message subject, and a message body.
Destination | A reusable location for an action. Supported locations are Amazon Chime, Email, Slack, or custom webhook.
Finding | An entry for an individual document found by a per document monitor query that contains the document ID, index name, and timestamp. Findings are stored in the Findings index: `.opensearch-alerting-finding*`.
Channel | A notification channel to use in an action. See [notifications]({{site.url}}{{site.baseurl}}/notifications-plugin/index) for more information.
## Per document monitors
Introduced 2.0
{: .label .label-purple }
The per query and per bucket monitors can only run a single query with one trigger condition. Per document monitors allow you to combine multiple query trigger conditions by adding a tag to the queries. Then you can add the tag as a single trigger condition instead of specifying a single query. The Alerting plugin processes the trigger conditions from all queries as a logical OR operation, so if any of the query conditions are met, it triggers an alert. Next, the Alerting plugin tells the Notifications plugin to send the notification to a channel.
The Alerting plugin also creates a list of document findings that contains metadata about which document matches each query. Security analytics can use the document findings data to keep track of and analyze the query data separately from the alert processes.
The Alerting API provides a document-level monitor that programmatically accomplishes the same function as the per document monitor in the OpenSearch Dashboards. To learn more, see [Document-level monitors]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/api/#document-level-monitors).
{: .note}
### Document findings
When a per document monitor executes a query that matches a document in an index, a finding is created. OpenSearch provides a Findings index: `.opensearch-alerting-finding*` that contains findings data for all per document monitor queries. You can search the findings index with the Alerting API search operation. To learn more, see [Search for monitor findings]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/api/#search-for-monitor-findings).
The following metadata is provided for each document finding entry:
* **Document** – The document ID and index name. For example: `Re5akdirhj3fl | test-logs-index`.
* **Query** – The query name that matched the document.
* **Time found** – The timestamp that indicates when the document was found during the runtime.
It is possible to configure an alert notification for each finding, however we don't recommend this unless rules are well defined to prevent a huge volume of findings in a high ingestion cluster.
---
## Create destinations
1. Choose **Alerting**, **Destinations**, **Add destination**.
1. Specify a name for the destination so that you can identify it later.
1. For **Type**, choose Slack, Amazon Chime, custom webhook, or [email](#email-as-a-destination).
For Email, refer to the [Email as a destination](#email-as-a-destination) section below. For all other types, specify the webhook URL. See the documentation for [Slack](https://api.slack.com/incoming-webhooks) and [Amazon Chime](https://docs.aws.amazon.com/chime/latest/ug/webhooks.html) to learn more about webhooks.
If you're using custom webhooks, you must specify more information: parameters and headers. For example, if your endpoint requires basic authentication, you might need to add a header with a key of `Authorization` and a value of `Basic <Base64-encoded-credential-string>`. You might also need to change `Content-Type` to whatever your webhook requires. Popular values are `application/json`, `application/xml`, and `text/plain`.
This information is stored in plain text in the OpenSearch cluster. We will improve this design in the future, but for now, the encoded credentials (which are neither encrypted nor hashed) might be visible to other OpenSearch users.
### Email as a destination
To send or receive an alert notification as an email, choose **Email** as the destination type. Next, add at least one sender and recipient. We recommend adding email groups if you want to notify more than a few people of an alert. You can configure senders and recipients using **Manage senders** and **Manage email groups**.
#### Manage senders
You need to specify an email account from which the Alerting plugin can send notifications.
To configure a sender email, do the following:
1. After you choose **Email** as the destination type, choose **Manage senders**.
1. Choose **Add sender**, **New sender** and enter a unique name.
1. Enter the email address, SMTP host (e.g. `smtp.gmail.com` for a Gmail account), and the port.
1. Choose an encryption method, or use the default value of **None**. However, most email providers require SSL or TLS, which require a username and password in OpenSearch keystore. Refer to [Authenticate sender account](#authenticate-sender-account) to learn more.
1. Choose **Save** to save the configuration and create the sender. You can create a sender even before you add your credentials to the OpenSearch keystore. However, you must [authenticate each sender account](#authenticate-sender-account) before you use the destination to send your alert.
You can reuse senders across many different destinations, but each destination only supports one sender.
#### Manage email groups or recipients
Use email groups to create and manage reusable lists of email addresses. For example, one alert might email the DevOps team, whereas another might email the executive team and the engineering team.
You can enter individual email addresses or an email group in the **Recipients** field.
1. After you choose **Email** as the destination type, choose **Manage email groups**. Then choose **Add email group**, **New email group**.
1. Enter a unique name.
1. For recipient emails, enter any number of email addresses.
1. Choose **Save**.
#### Authenticate sender account
If your email provider requires SSL or TLS, you must authenticate each sender account before you can send an email. Enter these credentials in the OpenSearch keystore using the CLI. Run the following commands (in your OpenSearch directory) to enter your username and password. The `<sender_name>` is the name you entered for **Sender** earlier.
```bash
./bin/opensearch-keystore add plugins.alerting.destination.email.<sender_name>.username
./bin/opensearch-keystore add plugins.alerting.destination.email.<sender_name>.password
```
Note: Keystore settings are node-specific. You must run these commands on each node.
{: .note}
To change or update your credentials (after you've added them to the keystore on every node), call the reload API to automatically update those credentials without restarting OpenSearch:
```json
POST _nodes/reload_secure_settings
{
"secure_settings_password": "1234"
}
```
---
## Create a monitor