Merge pull request #468 from opensearch-project/observability-tweaks
Observability language tweaks
This commit is contained in:
Keith Chan
GitHub
commit
da9033c6ab
|
|
@ -130,7 +130,7 @@ You can specify the following options in the HTTP request body:
|
|||
Option | Data Type | Description | Required
|
||||
:--- | :--- | :--- | :---
|
||||
enabled | Boolean | If true, the transform job is enabled at creation. | No
|
||||
continuous | Boolean | Specifies whether the transform job should be continuous. Continuous jobs execute every time they are scheduled according to the `schedule` option and run based off of newly transformed buckets and any new data added to source indexes. Non-continuous jobs execute only once. Default is false. | No
|
||||
continuous | Boolean | Specifies whether the transform job should be continuous. Continuous jobs execute every time they are scheduled according to the `schedule` field and run based off of newly transformed buckets, as well as any new data added to source indexes. Non-continuous jobs execute only once. Default is false. | No
|
||||
schedule | Object | The schedule the transform job runs on. | Yes
|
||||
start_time | Integer | The Unix epoch time of the transform job's start time. | Yes
|
||||
description | String | Describes the transform job. | No
|
||||
|
|
|
|||
|
|
@ -185,7 +185,7 @@ Whereas query-level monitors run your specified query and then check whether the
|
|||
|
||||
1. Add a trigger to your monitor.
|
||||
|
||||
---
|
||||
---
|
||||
## Create triggers
|
||||
|
||||
Steps to create a trigger differ depending on whether you chose **Visual editor**, **Extraction query editor**, or **Anomaly detector** when you created the monitor.
|
||||
|
|
@ -410,7 +410,7 @@ Deleted | Someone deleted the monitor or trigger associated with this alert whil
|
|||
|
||||
## Create cluster metrics monitor
|
||||
|
||||
In addition to monitoring conditions for indexes, the alerting plugin allows monitoring conditions on clusters. Alerts can be set by cluster metrics to watch for when:
|
||||
In addition to monitoring conditions for indexes, the alerting plugin allows monitoring conditions on clusters. Alerts can be set by cluster metrics to watch for when:
|
||||
|
||||
- The health of your cluster reaches a status of yellow or red.
|
||||
- Cluster-level metrics, such as CPU usage and JVM memory usage, reach specified thresholds.
|
||||
|
|
@ -420,16 +420,16 @@ In addition to monitoring conditions for indexes, the alerting plugin allows mon
|
|||
To create a cluster metrics monitor:
|
||||
|
||||
1. Select **Alerting** > **Monitors** > **Create monitor**.
|
||||
2. Select the **Per cluster metrics monitor** option.
|
||||
2. Select the **Per cluster metrics monitor** option.
|
||||
3. In the Query section, pick the **Request type** from the dropdown.
|
||||
4. (Optional) If you want to filter the API response to use only certain path parameters, enter those parameters under **Query parameters**. Most APIs that can be used to monitor cluster status support path parameters as described in their documentation (e.g., comma-separated lists of index names).
|
||||
5. In the Triggers section, indicate what conditions trigger an alert. The trigger condition autopopulates a painless ctx variable. For example, a cluster monitor watching for Cluster Stats uses the trigger condition `ctx.results[0].indices.count <= 0`, which triggers an alert based on the number of indexes returned by the query. For more specificity, add any additional painless conditions supported by the API. To see an example of the condition response, select **Preview condition response**.
|
||||
5. In the Triggers section, indicate what conditions trigger an alert. The trigger condition autopopulates a painless ctx variable. For example, a cluster monitor watching for Cluster Stats uses the trigger condition `ctx.results[0].indices.count <= 0`, which triggers an alert based on the number of indexes returned by the query. For more specificity, add any additional painless conditions supported by the API. To see an example of the condition response, select **Preview condition response**.
|
||||
6. In the Actions section, indicate how you want your users to be notified when a trigger condition is met.
|
||||
7. Select **Create**. Your new monitor appears in the **Monitors** list.
|
||||
|
||||
### Supported APIs
|
||||
|
||||
Trigger conditions use responses from the following APIs. Most APIs that can be used to monitor cluster status support path parameters as described in their documentation (e.g., comma-separated lists of index names). However, they do not support query parameters.
|
||||
Trigger conditions use responses from the following APIs. Most APIs that can be used to monitor cluster status support path parameters as described in their documentation (e.g., comma-separated lists of index names). However, they do not support query parameters.
|
||||
|
||||
1. [_cluster/health]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cluster-health/)
|
||||
2. [_cluster/stats]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cluster-stats/)
|
||||
|
|
@ -442,16 +442,16 @@ Trigger conditions use responses from the following APIs. Most APIs that can be
|
|||
|
||||
### Restrict API fields
|
||||
|
||||
If you want to hide fields from the API response that you do not want exposed for alerting, reconfigure the [supported_json_payloads.json](https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/resources/org/opensearch/alerting/settings/supported_json_payloads.json) inside your alerting plugin. The file functions as an allow list for the API fields you want to use in an alert. By default, all APIs and their parameters can be used for monitors and trigger conditions.
|
||||
If you want to hide fields from the API response that you do not want exposed for alerting, reconfigure the [supported_json_payloads.json](https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/resources/org/opensearch/alerting/settings/supported_json_payloads.json) inside your alerting plugin. The file functions as an allow list for the API fields you want to use in an alert. By default, all APIs and their parameters can be used for monitors and trigger conditions.
|
||||
|
||||
However, you can modify the file so that cluster metric monitors can only be created for APIs referenced. Furthermore, only fields referenced in the supported files can create trigger conditions. This `supported_json_payloads.json` allows for a cluster metrics monitor to be created for the `_cluster/stats` API, and triggers conditions for the `indices.shards.total` and `indices.shards.index.shards.min` fields.
|
||||
However, you can modify the file so that cluster metric monitors can only be created for APIs referenced. Furthermore, only fields referenced in the supported files can create trigger conditions. This `supported_json_payloads.json` allows for a cluster metrics monitor to be created for the `_cluster/stats` API, and triggers conditions for the `indices.shards.total` and `indices.shards.index.shards.min` fields.
|
||||
|
||||
```json
|
||||
"/_cluster/stats": {
|
||||
"indices": [
|
||||
"shards.total",
|
||||
"shards.index.shards.min"
|
||||
]
|
||||
"indices": [
|
||||
"shards.total",
|
||||
"shards.index.shards.min"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -465,50 +465,50 @@ In this example, a JSON object creates a trigger that sends an alert when the Cl
|
|||
|
||||
```json
|
||||
{
|
||||
"name": "Cluster Health Monitor",
|
||||
"type": "monitor",
|
||||
"monitor_type": "query_level_monitor",
|
||||
"enabled": true,
|
||||
"schedule": {
|
||||
"period": {
|
||||
"unit": "MINUTES",
|
||||
"interval": 1
|
||||
"name": "Cluster Health Monitor",
|
||||
"type": "monitor",
|
||||
"monitor_type": "query_level_monitor",
|
||||
"enabled": true,
|
||||
"schedule": {
|
||||
"period": {
|
||||
"unit": "MINUTES",
|
||||
"interval": 1
|
||||
}
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"uri": {
|
||||
"api_type": "CLUSTER_HEALTH",
|
||||
"path": "_cluster/health/",
|
||||
"path_params": "",
|
||||
"url": "http://localhost:9200/_cluster/health/"
|
||||
}
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"uri": {
|
||||
"api_type": "CLUSTER_HEALTH",
|
||||
"path": "_cluster/health/",
|
||||
"path_params": "",
|
||||
"url": "http://localhost:9200/_cluster/health/"
|
||||
}
|
||||
}
|
||||
],
|
||||
"triggers": [
|
||||
{
|
||||
"query_level_trigger": {
|
||||
"id": "Tf_L_nwBti6R6Bm-18qC",
|
||||
"name": "Yellow status trigger",
|
||||
"severity": "1",
|
||||
"condition": {
|
||||
"script": {
|
||||
"source": "ctx.results[0].status == \"yellow\"",
|
||||
"lang": "painless"
|
||||
}
|
||||
},
|
||||
"actions": []
|
||||
}
|
||||
],
|
||||
"triggers": [
|
||||
{
|
||||
"query_level_trigger": {
|
||||
"id": "Tf_L_nwBti6R6Bm-18qC",
|
||||
"name": "Yellow status trigger",
|
||||
"severity": "1",
|
||||
"condition": {
|
||||
"script": {
|
||||
"source": "ctx.results[0].status == \"yellow\"",
|
||||
"lang": "painless"
|
||||
}
|
||||
},
|
||||
"actions": []
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
See [trigger variables](#trigger-variables), for more painless ctx options.
|
||||
See [trigger variables](#trigger-variables) for more painless ctx options.
|
||||
|
||||
### Limitations
|
||||
|
||||
Currently, the cluster metrics monitor has the following limitations:
|
||||
Currently, the cluster metrics monitor has the following limitations:
|
||||
|
||||
- You cannot create monitors for remote clusters.
|
||||
- The OpenSearch cluster must be in a state where an index's conditions can be monitored and actions can be executed against the index.
|
||||
|
|
|
|||
|
|
@ -6,17 +6,18 @@ nav_order: 80
|
|||
|
||||
# Application analytics
|
||||
|
||||
Application analytics can now create custom Observability Applications to view the availability status of all their systems, where log events can be combined with trace and metric data, into a single view of system health empowering developers and IT Ops to resolve issues faster and with fewer escalations.Application analytics in Observability is where you can select logs, traces, and metrics to be part of an Observability Application that can be monitored for overall health and visualized on one page. This allows you to quickly pivot between logs, traces, and metrics to dig into the source of any issues. Many teams use similar logs and similar traces and need a structured view.
|
||||
You can use application analytics to create custom observability applications to view the availability status of your systems, where you can combine log events with trace and metric data into a single view of overall system health. This lets you quickly pivot between logs, traces, and metrics to dig into the source of any issues.
|
||||
|
||||
## Get started with application analytics
|
||||
|
||||
To get started, choose **Observability** in OpenSearch Dashboards, and then choose **Application analytics**.
|
||||
To get started, choose **Observability** in OpenSearch Dashboards, and then choose **Application analytics**.
|
||||
|
||||
### Create an application
|
||||
|
||||
1. Choose **Create application**
|
||||
1. Enter a name for your application and optionally add a description
|
||||
1. Do at least one of the following
|
||||
1. Choose **Create application**.
|
||||
2. Enter a name for your application and optionally add a description.
|
||||
3. Do at least one of the following:
|
||||
|
||||
- Use [PPL]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) to specify the base query.
|
||||
|
||||
You can't change the base query after the application is created.
|
||||
|
|
@ -24,40 +25,41 @@ You can't change the base query after the application is created.
|
|||
|
||||
- Select services & entities from the dropdown or the service map.
|
||||
- Select trace groups from the dropdown or the table.
|
||||
1. Choose **Create**.
|
||||
|
||||
4\. Choose **Create**.
|
||||
|
||||
### Create a visualization
|
||||
|
||||
1. Choose the **Log Events** tab
|
||||
1. Use [PPL]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) to build upon your base query
|
||||
1. Move to the **Visualizations** tab to see your visualizations
|
||||
1. Choose the **Log Events** tab.
|
||||
1. Use [PPL]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) to build upon your base query.
|
||||
1. Choose the **Visualizations** tab to see your visualizations.
|
||||
1. Expand the **Save** dropdown menu, enter a name for your visualization, then choose **Save**.
|
||||
|
||||
To see your visualizations, move to the **Panel** tab.
|
||||
To see your visualizations, choose the **Panel** tab.
|
||||
|
||||
### Configure availability
|
||||
|
||||
Availability is the status of your application determined by availability levels set on a [time series metric]({{site.url}}{{site.baseurl}}/observability-plugin/app-analytics/#time-series-metric).
|
||||
Availability is the status of your application determined by availability levels set on a [time series metric]({{site.url}}{{site.baseurl}}/observability-plugin/app-analytics/#time-series-metric).
|
||||
|
||||
To create an availability level you must configure the following
|
||||
To create an availability level, you must configure the following:
|
||||
- color: The color of the availability badge on the home page.
|
||||
- name: The text in the availability badge on the home page.
|
||||
- expression: Comparison operator to determine the availability.
|
||||
- expression: Comparison operator to determine the availability.
|
||||
- value: Value to use when calculating availability.
|
||||
|
||||
|
||||
|
||||
By default, Application analytics shows results from the last 24 hours of your data. To see data from a different timeframe, use the date and time selector.
|
||||
By default, application analytics shows results from the last 24 hours of your data. To see data from a different time frame, use the date and time selector.
|
||||
|
||||
#### Time series metric
|
||||
|
||||
A time series metric is any visualization that has a query that spans over a timestamp and is a line chart. Users can use the power of the PPL language to define arbitrary conditions on their logs to create a visualization over time.
|
||||
A time series metric is any visualization that has a query that spans over a timestamp and is a line chart. You can then use PPL to define arbitrary conditions on your logs to create a visualization over time.
|
||||
|
||||
##### Example
|
||||
```
|
||||
source = <index_name> | ... | ... | stats ... by span(<timestamp_field>, 1h)
|
||||
```
|
||||
|
||||
Users can then choose **Line** in visualization configurations to create a time series metric.
|
||||
Choose **Line** in visualization configurations to create a time series metric.
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -34,22 +34,24 @@ To save a visualization, expand the save dropdown menu next to **Refresh**, ente
|
|||
|
||||
## View logs
|
||||
|
||||
Users have several ways to view their logs.
|
||||
The following are methods you can use to view logs.
|
||||
|
||||
### Trace log correlation
|
||||
|
||||
Users who regularly track events across applications have the capability to correlate logs and traces. To view the correlation, users have to index the traces as per the Open Telemetry standards (similar to Trace analytics). Once users add a **TraceId** field to their logs, they can view the trace information that corresponds to the same execution context as the log.
|
||||
If you regularly track events across applications, you can correlate logs and traces. To view the correlation, you have to index the traces according to Open Telemetry standards (similar to trace analytics). Once you add a `TraceId` field to your logs, you can view the correlated trace information in the event explorer log details. This method lets you correlate logs and traces that correspond to the same execution context.
|
||||
|
||||
|
||||
|
||||
### Surrounding events
|
||||
### View surrounding events
|
||||
|
||||
Users who want to know more about a log event that they are looking at can select **View surrounding events** to get a bigger picture of what was happening around the time of interest.
|
||||
If you want to know more about a log event you're looking at, you can select **View surrounding events** to get a bigger picture of what was happening around the time of interest.
|
||||
|
||||
|
||||
|
||||
### Live Tail
|
||||
### Live tail
|
||||
|
||||
Users watching a live event take place can now configure the interval in which content is refreshed saving the hassle of manually refreshing. This provides users with a faster debugging experience, and allows them to monitor their logs in real-time. Users also have the convenience of choosing the interval at which live tail should update. As this feature mimics the "tail -f" command in command line, it only retrieves the most recent live logs by possibly eliminating a heavy load of live logs.
|
||||
If you prefer watching events happen live, you can configure an interval so event analytics automatically refreshes the content. Live tail lets you stream logs live to OpenSearch observability event analytics based on the provided PPL query, as well as provide rich functionality such as filters. Doing so improves your debugging experience and lets you monitor your logs in real-time without having to manually refresh.
|
||||
|
||||
|
||||
You can also choose intervals and switch between them to dictate how often live tail should stream live logs. This feature is similar to the CLI's `tail -f` command in that it only retrieves the most recent live logs by possibly eliminating a large portion of live logs. Live tail also provides you with the total count of live logs received by OpenSearch during the live stream, which you can use to better understand the incoming traffic.
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue