Merge branch 'main' of github.com:opensearch-project/documentation-website into ml-commons
Signed-off-by: Naarcha-AWS naarcha@amazon.com
This commit is contained in:
Naarcha-AWS
commit
69ebe09b78
|
|
@ -5,10 +5,10 @@ baseurl: "/docs/latest" # the subpath of your site, e.g. /blog
|
|||
url: "https://opensearch.org" # the base hostname & protocol for your site, e.g. http://example.com
|
||||
permalink: /:path/
|
||||
|
||||
opensearch_version: 1.2.4
|
||||
opensearch_dashboards_version: 1.2.0
|
||||
opensearch_major_minor_version: 1.2
|
||||
lucene_version: 8_9_0
|
||||
opensearch_version: 1.3.0
|
||||
opensearch_dashboards_version: 1.3.0
|
||||
opensearch_major_minor_version: 1.3
|
||||
lucene_version: 8_10_1
|
||||
|
||||
# Build settings
|
||||
markdown: kramdown
|
||||
|
|
|
|||
|
|
@ -54,7 +54,7 @@ To start using your own branding elements in OpenSearch Dashboards, first uncomm
|
|||
|
||||
Add the URLs you want to use as branding elements to the appropriate setting. Valid image types are `SVG`, `PNG`, and `GIF`.
|
||||
|
||||
Customization of dark mode Dashboards is also available, but you first must supply a valid link to `defaultUrl`, and then link to your preferred image with `darkModeUrl`. You are not required to customize all branding elements, so if you wanted to, it's perfectly valid to change just the logo. Leave unchanged elements as commented.
|
||||
Customization of dark mode Dashboards is also available, but you first must supply a valid link to `defaultUrl`, and then link to your preferred image with `darkModeUrl`. If you don't provide a `darkModeUrl` link, then Dashboards uses the provided `defaultUrl` element for dark mode. You are not required to customize all branding elements, so if you wanted to, it's perfectly valid to change just the logo or any other element. Leave unchanged elements as commented.
|
||||
|
||||
The following example demonstrates how to use `SVG` files as logos but leaves the other elements as defaults.
|
||||
|
||||
|
|
@ -72,17 +72,17 @@ logo:
|
|||
applicationTitle: "My custom application"
|
||||
```
|
||||
|
||||
We recommend linking to images that are hosted on a web server, but if you really want to use locally hosted images, save your images inside `src/core/server/core_app/assets`, and then configure `opensearch_dashboards.yml`. You can access locally stored images through the `ui` folder.
|
||||
We recommend linking to images that are hosted on a web server, but if you really want to use locally hosted images, save your images inside `assets`, and then configure `opensearch_dashboards.yml` to use the correct paths. You can access locally stored images through the `ui/assets` folder.
|
||||
|
||||
The following example assumes the default port of 5601 that Dashboards uses and demonstrates how to link to locally stored images.
|
||||
|
||||
```yml
|
||||
logo:
|
||||
defaultUrl: "https://localhost:5601/ui/my-own-image.svg"
|
||||
darkModeUrl: "https://localhost:5601/ui/my-own-image.svg"
|
||||
defaultUrl: "https://localhost:5601/ui/assets/my-own-image.svg"
|
||||
darkModeUrl: "https://localhost:5601/ui/assets/dark-mode-my-own-image.svg"
|
||||
mark:
|
||||
defaultUrl: "https://localhost:5601/ui/my-own-image2.svg"
|
||||
darkModeUrl: "https://localhost:5601/ui/my-own-image2.svg"
|
||||
defaultUrl: "https://localhost:5601/ui/assets/my-own-image2.svg"
|
||||
darkModeUrl: "https://localhost:5601/ui/assets/dark-mode-my-own-image2.svg"
|
||||
# loadingLogo:
|
||||
# defaultUrl: ""
|
||||
# darkModeUrl: ""
|
||||
|
|
@ -90,11 +90,9 @@ mark:
|
|||
applicationTitle: "My custom application"
|
||||
```
|
||||
|
||||
We don't recommend this workaround because new versions of Dashboards would revert all customized elements back to default branding elements, and you would have to re-upload your assets to access them again.
|
||||
|
||||
## Sample configuration
|
||||
|
||||
The following configuration enables the security plugin within OpenSearch Dashboards and uses custom branding elements to replace the OpenSearch logo and application title.
|
||||
The following configuration enables the security plugin and SSL within OpenSearch Dashboards and uses custom branding elements to replace the OpenSearch logo and application title.
|
||||
|
||||
```yml
|
||||
server.host: "0"
|
||||
|
|
@ -103,6 +101,9 @@ opensearch.ssl.verificationMode: none
|
|||
opensearch.username: "kibanaserver"
|
||||
opensearch.password: "kibanaserver"
|
||||
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]
|
||||
#server.ssl.enabled: true
|
||||
#server.ssl.certificate: /path/to/your/server/certificate
|
||||
#server.ssl.key: /path/to/your/server/key
|
||||
|
||||
opensearch_security.multitenancy.enabled: true
|
||||
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
|
||||
|
|
@ -113,7 +114,7 @@ opensearch_security.cookie.secure: false
|
|||
opensearchDashboards.branding:
|
||||
logo:
|
||||
defaultUrl: "https://example.com/sample.svg"
|
||||
darkModeUrl: "https://example.com/sample.svg"
|
||||
darkModeUrl: "https://example.com/dark-mode-sample.svg"
|
||||
# mark:
|
||||
# defaultUrl: ""
|
||||
# darkModeUrl: ""
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
message: "OpenSearch 1.2.0 is live 🍾 Try the new observability interface, branding customizer, and more!](/downloads.html)"
|
||||
message: "[OpenSearch 1.3.0 is live 🍾 Try the new ML commons plugin, new observability features, and much, much more!](/downloads.html)"
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
{
|
||||
"current": "1.2",
|
||||
"current": "1.3",
|
||||
"past": [
|
||||
"1.2",
|
||||
"1.1",
|
||||
"1.0"
|
||||
]
|
||||
|
|
|
|||
|
|
@ -9,16 +9,16 @@ has_toc: false
|
|||
|
||||
# Index transforms
|
||||
|
||||
Whereas index rollup jobs let you reduce data granularity by rolling up old data into condensed indices, transform jobs let you create a different, summarized view of your data centered around certain fields, so you can visualize or analyze the data in different ways.
|
||||
Whereas index rollup jobs let you reduce data granularity by rolling up old data into condensed indexes, transform jobs let you create a different, summarized view of your data centered around certain fields, so you can visualize or analyze the data in different ways.
|
||||
|
||||
For example, suppose that you have airline data that’s scattered across multiple fields and categories, and you want to view a summary of the data that’s organized by airline, quarter, and then price. You can use a transform job to create a new, summarized index that’s organized by those specific categories.
|
||||
|
||||
You can use transform jobs in two ways:
|
||||
|
||||
1. Use the OpenSearch Dashboards UI to specify the index you want to transform and any optional data filters you want to use to filter the original index. Then select the fields you want to transform and the aggregations to use in the transformation. Finally, define a schedule for your job to follow.
|
||||
2. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow.
|
||||
1. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow.
|
||||
|
||||
OpenSearch Dashboards provides a detailed summary of the jobs you created and their relevant information, such as associated indices and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards.
|
||||
OpenSearch Dashboards provides a detailed summary of the jobs you created and their relevant information, such as associated indexes and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards.
|
||||
|
||||
Your use cases will help you decide which method to use to create transform jobs.
|
||||
|
||||
|
|
@ -26,7 +26,7 @@ Your use cases will help you decide which method to use to create transform jobs
|
|||
|
||||
If you don't have any data in your cluster, you can use the sample flight data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose **Index Management**. Select **Transform Jobs**, and choose **Create Transform Job**.
|
||||
|
||||
### Step 1: Choose indices
|
||||
### Step 1: Choose indexes
|
||||
|
||||
1. In the **Job name and description** section, specify a name and an optional description for your job.
|
||||
2. In the **Indices** section, select the source and target index. You can either select an existing target index or create a new one by entering a name for your new index. If you want to transform just a subset of your source index, choose **Edit data filter**, and use the OpenSearch query DSL to specify a subset of your source index. For more information about the OpenSearch query DSL, see [query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/).
|
||||
|
|
@ -34,7 +34,7 @@ If you don't have any data in your cluster, you can use the sample flight data w
|
|||
|
||||
### Step 2: Select fields to transform
|
||||
|
||||
After specifying the indices, you can select the fields you want to use in your transform job, as well as whether to use groupings or aggregations.
|
||||
After specifying the indexes, you can select the fields you want to use in your transform job, as well as whether to use groupings or aggregations.
|
||||
|
||||
You can use groupings to place your data into separate buckets in your transformed index. For example, if you want to group all of the airport destinations within the sample flight data, you can group the `DestAirportID` field into a target field of `DestAirportID_terms` field, and you can find the grouped airport IDs in your transformed index after the transform job finishes.
|
||||
|
||||
|
|
@ -44,16 +44,17 @@ On the other hand, aggregations let you perform simple calculations. For example
|
|||
|
||||
Currently, transform jobs support histogram, date_histogram, and terms groupings. For more information about groupings, see [Bucket Aggregations]({{site.url}}{{site.baseurl}}/opensearch/bucket-agg/). In terms of aggregations, you can select from `sum`, `avg`, `max`, `min`, `value_count`, `percentiles`, and `scripted_metric`. For more information about aggregations, see [Metric Aggregations]({{site.url}}{{site.baseurl}}/opensearch/metric-agg/).
|
||||
|
||||
2. Repeat step 1 for any other fields that you want to transform.
|
||||
3. After selecting the fields that you want to transform and verifying the transformation, choose **Next**.
|
||||
1. Repeat step 1 for any other fields that you want to transform.
|
||||
1. After selecting the fields that you want to transform and verifying the transformation, choose **Next**.
|
||||
|
||||
### Step 3: Specify a schedule
|
||||
|
||||
You can configure transform jobs to run once or multiple times on a schedule. Transform jobs are enabled by default.
|
||||
|
||||
1. For **transformation execution interval**, specify a transform interval in minutes, hours, or days.
|
||||
2. Under **Advanced**, specify an optional amount for **Pages per execution**. A larger number means more data is processed in each search request, but also uses more memory and causes higher latency. Exceeding allowed memory limits can cause exceptions and errors to occur.
|
||||
3. Choose **Next**.
|
||||
1. Choose whether the job should be **continuous**. Continuous jobs execute at each **transform execution interval** and incrementally transform newly modified buckets, which can include new data added to the source indexes. Non-continuous jobs execute only once.
|
||||
1. For **transformation execution interval**, specify a transform interval in minutes, hours, or days. This interval dicatates how often continuous jobs should execute, and non-continuous jobs execute once after the interval elapses.
|
||||
1. Under **Advanced**, specify an optional amount for **Pages per execution**. A larger number means more data is processed in each search request, but also uses more memory and causes higher latency. Exceeding allowed memory limits can cause exceptions and errors to occur.
|
||||
1. Choose **Next**.
|
||||
|
||||
### Step 4: Review and confirm details
|
||||
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ PUT _plugins/_transform/<transform_id>
|
|||
{
|
||||
"transform": {
|
||||
"enabled": true,
|
||||
"continuous": true,
|
||||
"schedule": {
|
||||
"interval": {
|
||||
"period": 1,
|
||||
|
|
@ -78,6 +79,7 @@ PUT _plugins/_transform/<transform_id>
|
|||
"transform": {
|
||||
"transform_id": "sample",
|
||||
"schema_version": 7,
|
||||
"continuous": true,
|
||||
"schedule": {
|
||||
"interval": {
|
||||
"start_time": 1621467964243,
|
||||
|
|
@ -128,6 +130,7 @@ You can specify the following options in the HTTP request body:
|
|||
Option | Data Type | Description | Required
|
||||
:--- | :--- | :--- | :---
|
||||
enabled | Boolean | If true, the transform job is enabled at creation. | No
|
||||
continuous | Boolean | Specifies whether the transform job should be continuous. Continuous jobs execute every time they are scheduled according to the `schedule` option and run based off of newly transformed buckets and any new data added to source indexes. Non-continuous jobs execute only once. Default is false. | No
|
||||
schedule | Object | The schedule the transform job runs on. | Yes
|
||||
start_time | Integer | The Unix epoch time of the transform job's start time. | Yes
|
||||
description | String | Describes the transform job. | No
|
||||
|
|
@ -291,6 +294,7 @@ GET _plugins/_transform/<transform_id>
|
|||
"transform": {
|
||||
"transform_id": "sample",
|
||||
"schema_version": 7,
|
||||
"continuous": true,
|
||||
"schedule": {
|
||||
"interval": {
|
||||
"start_time": 1621467964243,
|
||||
|
|
@ -358,6 +362,7 @@ GET _plugins/_transform/
|
|||
"transform": {
|
||||
"transform_id": "sample",
|
||||
"schema_version": 7,
|
||||
"continuous": true,
|
||||
"schedule": {
|
||||
"interval": {
|
||||
"start_time": 1621467964243,
|
||||
|
|
@ -595,6 +600,12 @@ GET _plugins/_transform/<transform_id>/_explain
|
|||
"sample": {
|
||||
"metadata_id": "PzmjweME5xbgkenl9UpsYw",
|
||||
"transform_metadata": {
|
||||
"continuous_stats": {
|
||||
"last_timestamp": 1621883525672,
|
||||
"documents_behind": {
|
||||
"sample_index": 72
|
||||
}
|
||||
},
|
||||
"transform_id": "sample",
|
||||
"last_updated_at": 1621883525873,
|
||||
"status": "finished",
|
||||
|
|
@ -702,7 +713,7 @@ POST _plugins/_transform/_preview
|
|||
Introduced 1.0
|
||||
{: .label .label-purple }
|
||||
|
||||
Deletes a transform job. This operation does not delete the source or target indices.
|
||||
Deletes a transform job. This operation does not delete the source or target indexes.
|
||||
|
||||
**Sample Request**
|
||||
|
||||
|
|
|
|||
|
|
@ -176,6 +176,8 @@ Introduced 1.0
|
|||
|
||||
Updates a policy. Use the `seq_no` and `primary_term` parameters to update an existing policy. If these numbers don't match the existing policy or the policy doesn't exist, ISM throws an error.
|
||||
|
||||
It's possible that the policy currently applied to your index isn't the most up-to-date policy available. To see what policy is currently applied to your index, see [Explain index]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#explain-index). To get the most up-to-date version of a policy, see [Get policy]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#get-policy).
|
||||
|
||||
#### Request
|
||||
|
||||
```json
|
||||
|
|
@ -475,6 +477,40 @@ GET _plugins/_ism/explain/index_1
|
|||
}
|
||||
```
|
||||
|
||||
Optionally, you can add the `show_policy` parameter to your request's path to get the policy that is currently applied to your index, which is useful for seeing whether the policy applied to your index is the latest one. To get the most up-to-date policy, see [Get Policy API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#get-policy).
|
||||
|
||||
#### Request
|
||||
|
||||
```json
|
||||
GET _plugins/_ism/explain/index_1?show_policy=true
|
||||
```
|
||||
|
||||
#### Sample response
|
||||
|
||||
```json
|
||||
{
|
||||
"index_1": {
|
||||
"index.plugins.index_state_management.policy_id": "sample-policy",
|
||||
"index.opendistro.index_state_management.policy_id": "sample-policy",
|
||||
"index": "index_1",
|
||||
"index_uuid": "gCFlS_zcTdih8xyxf3jQ-A",
|
||||
"policy_id": "sample-policy",
|
||||
"enabled": true,
|
||||
"policy": {
|
||||
"policy_id": "sample-policy",
|
||||
"description": "ingesting logs",
|
||||
"last_updated_time": 1647284980148,
|
||||
"schema_version": 13,
|
||||
"error_notification": null,
|
||||
"default_state": "ingest",
|
||||
"states": [...],
|
||||
"ism_template": null
|
||||
}
|
||||
},
|
||||
"total_managed_indices": 1
|
||||
}
|
||||
```
|
||||
|
||||
The `plugins.index_state_management.policy_id` setting is deprecated starting from ODFE version 1.13.0. We retain this field in the response API for consistency.
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -322,7 +322,7 @@ You can specify the following options.
|
|||
Options | Description | Type | Required
|
||||
:--- | :--- |:--- |:--- |
|
||||
`name` | The name of the detector. | `string` | Yes
|
||||
`description` | A description of the detector. | `string` | Yes
|
||||
`description` | A description of the detector. | `string` | No
|
||||
`time_field` | The name of the time field. | `string` | Yes
|
||||
`indices` | A list of indices to use as the data source. | `list` | Yes
|
||||
`feature_attributes` | Specify a `feature_name`, set the `enabled` parameter to `true`, and specify an aggregation query. | `list` | Yes
|
||||
|
|
@ -339,10 +339,27 @@ Introduced 1.2
|
|||
|
||||
Returns whether the detector configuration has any issues that might prevent OpenSearch from creating the detector.
|
||||
|
||||
You can use the validate API to identify issues in your detector configuration before creating the detector.
|
||||
You can use the validate detector API operation to identify issues in your detector configuration before creating the detector.
|
||||
|
||||
The request body consists of the detector configuration and follows the same format as the request body of the [create detector API]({{site.url}}{{site.baseurl}}/monitoring-plugins/ad/api#create-anomaly-detector).
|
||||
|
||||
You have the following validation options:
|
||||
|
||||
- Only validate against the detector configuration and find any issues that would completely block detector creation:
|
||||
|
||||
```
|
||||
POST _plugins/_anomaly_detection/detectors/_validate
|
||||
POST _plugins/_anomaly_detection/detectors/_validate/detector
|
||||
```
|
||||
|
||||
- Validate against the source data to see how likely the detector would complete model training.
|
||||
|
||||
```
|
||||
POST _plugins/_anomaly_detection/detectors/_validate/model
|
||||
```
|
||||
|
||||
Responses from this API operation return either blocking issues as detector type responses or a response indicating a field that could be revised to increase likelihood of model training completing successfully. Model type issues don’t need to be fixed for detector creation to succeed, but the detector would likely not train successfully if they aren’t addressed.
|
||||
|
||||
#### Request
|
||||
|
||||
```json
|
||||
|
|
@ -398,7 +415,7 @@ POST _plugins/_anomaly_detection/detectors/_validate/detector
|
|||
}
|
||||
```
|
||||
|
||||
If the validate API doesn’t find any issue in the detector configuration, it returns an empty response:
|
||||
If the validate detector API doesn’t find any issue in the detector configuration, it returns an empty response:
|
||||
|
||||
#### Sample response
|
||||
|
||||
|
|
@ -406,7 +423,7 @@ If the validate API doesn’t find any issue in the detector configuration, it r
|
|||
{}
|
||||
```
|
||||
|
||||
If the validate API finds an issue, it returns a message explaining what's wrong with the configuration. In this example, the feature query aggregates over a field that doesn’t exist in the data source:
|
||||
If the validate detector API finds an issue, it returns a message explaining what's wrong with the configuration. In this example, the feature query aggregates over a field that doesn’t exist in the data source:
|
||||
|
||||
#### Sample response
|
||||
|
||||
|
|
@ -422,6 +439,95 @@ If the validate API finds an issue, it returns a message explaining what's wrong
|
|||
}
|
||||
}
|
||||
```
|
||||
|
||||
The following request validates against the source data to see if model training might succeed. In this example, the data is ingested at a rate of every 5 minutes, and detector interval is set to 1 minute.
|
||||
|
||||
```json
|
||||
POST _plugins/_anomaly_detection/detectors/_validate/model
|
||||
{
|
||||
"name": "test-detector",
|
||||
"description": "Test detector",
|
||||
"time_field": "timestamp",
|
||||
"indices": [
|
||||
"server_log*"
|
||||
],
|
||||
"feature_attributes": [
|
||||
{
|
||||
"feature_name": "test",
|
||||
"feature_enabled": true,
|
||||
"aggregation_query": {
|
||||
"test": {
|
||||
"sum": {
|
||||
"field": "value"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"filter_query": {
|
||||
"bool": {
|
||||
"filter": [
|
||||
{
|
||||
"range": {
|
||||
"value": {
|
||||
"gt": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"adjust_pure_negative": true,
|
||||
"boost": 1
|
||||
}
|
||||
},
|
||||
"detection_interval": {
|
||||
"period": {
|
||||
"interval": 1,
|
||||
"unit": "Minutes"
|
||||
}
|
||||
},
|
||||
"window_delay": {
|
||||
"period": {
|
||||
"interval": 1,
|
||||
"unit": "Minutes"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If the validate detector API finds areas of improvement with your configuration, it returns a response with suggestions about how you can change your configuration to improve model training.
|
||||
|
||||
#### Sample Responses
|
||||
|
||||
In this example, the validate detector API returns a response indicating that changing the detector interval length to at least four minutes can increase the chances of successful model training.
|
||||
|
||||
```json
|
||||
{
|
||||
"model": {
|
||||
"detection_interval": {
|
||||
"message": "The selected detector interval might collect sparse data. Consider changing interval length to: 4",
|
||||
"suggested_value": {
|
||||
"period": {
|
||||
"interval": 4,
|
||||
"unit": "Minutes"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Another response might indicate that you can change `filter_query` (data filter) because the currently filtered data is too sparse for the model to train correctly, which can happen because the index is also ingesting data that falls outside the chosen filter. Using another `filter_query` can make your data more dense.
|
||||
|
||||
```json
|
||||
{
|
||||
"model": {
|
||||
"filter_query": {
|
||||
"message": "Data is too sparse after data filter is applied. Consider changing the data filter"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Get detector
|
||||
|
|
|
|||
|
|
@ -9,7 +9,6 @@ nav_order: 15
|
|||
|
||||
Use the alerting API to programmatically manage monitors and alerts.
|
||||
|
||||
|
||||
---
|
||||
|
||||
#### Table of contents
|
||||
|
|
|
|||
|
|
@ -185,8 +185,7 @@ Whereas query-level monitors run your specified query and then check whether the
|
|||
|
||||
1. Add a trigger to your monitor.
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
## Create triggers
|
||||
|
||||
Steps to create a trigger differ depending on whether you chose **Visual editor**, **Extraction query editor**, or **Anomaly detector** when you created the monitor.
|
||||
|
|
@ -406,3 +405,112 @@ Acknowledged | Someone has acknowledged the alert, but not fixed the root cause.
|
|||
Completed | The alert is no longer ongoing. Alerts enter this state after the corresponding trigger evaluates to false.
|
||||
Error | An error occurred while executing the trigger---usually the result of a a bad trigger or destination.
|
||||
Deleted | Someone deleted the monitor or trigger associated with this alert while the alert was ongoing.
|
||||
|
||||
---
|
||||
|
||||
## Create cluster metrics monitor
|
||||
|
||||
In addition to monitoring conditions for indexes, the alerting plugin allows monitoring conditions on clusters. Alerts can be set by cluster metrics to watch for when:
|
||||
|
||||
- The health of your cluster reaches a status of yellow or red.
|
||||
- CPU usages reaches a specified threshold.
|
||||
- JVM memory usage reaches a specified threshold.
|
||||
- The total number of documents stores reaches a specified amount.
|
||||
|
||||
To create a cluster metrics monitor:
|
||||
|
||||
1. Select **Alerting** > **Monitors** > **Create monitor**.
|
||||
2. Select the **Per cluster metrics monitor** option.
|
||||
3. In the Query section, pick the **Request type** from the dropdown.
|
||||
4. (Optional) If you want to filter the API response to use only certain path parameters, enter those parameters under **Query parameters**. Most APIs that can be used to monitor cluster status support path parameters as described in their documentation (e.g., comma-separated lists of index names).
|
||||
5. In the Triggers section, indicate what conditions trigger an alert. The trigger condition autopopulates a painless ctx variable. For example, a cluster monitor watching for Cluster Stats uses the trigger condition `ctx.results[0].indices.count <= 0`, which triggers an alert based on the number of indexes returned by the query. For more specificity, add any additional painless conditions supported by the API. To see an example of the condition response, select **Preview condition response**.
|
||||
6. In the Actions section, indicate how you want your users to be notified when a trigger condition is met.
|
||||
7. Select **Create**. Your new monitor appears in the **Monitors** list.
|
||||
|
||||
### Supported APIs
|
||||
|
||||
Trigger conditions use responses from the following APIs. Most APIs that can be used to monitor cluster status support path parameters as described in their documentation (e.g., comma-separated lists of index names). However, they do not support query parameters.
|
||||
|
||||
1. [_cluster/health]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cluster-health/)
|
||||
2. [_cluster/stats]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cluster-stats/
|
||||
3. [_cluster/settings]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cluster-settings/)
|
||||
4. [_nodes/stats]({{site.url}}{{site.baseurl}}/opensearch/popular-api/#get-node-statistics)
|
||||
5. _[cat/pending_tasks]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cat/cat-pending-tasks/)
|
||||
6. [_cat/recovery]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cat/cat-recovery/
|
||||
7. [_cat/snapshots]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cat/cat-snapshots/)
|
||||
8. [_cat/tasks]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cat/cat-tasks/)
|
||||
|
||||
### Restrict API fields
|
||||
|
||||
If you want to hide fields from the API response that you do not want exposed for alerting, reconfigure the [supported_json_payloads.json](https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/resources/org/opensearch/alerting/settings/supported_json_payloads.json) inside your alerting plugin. The file functions as an allow list for the API fields you want to use in an alert. By default, all APIs and their parameters can be used for monitors and trigger conditions.
|
||||
|
||||
However, you can modify the file so that cluster metric monitors can only be created for APIs referenced. Furthermore, only fields referenced in the supported files can create trigger conditions. This `supported_json_payloads.json` allows for a cluster metrics monitor to be created for the `_cluster/stats` API, and triggers conditions for the `indices.shards.total` and `indices.shards.index.shards.min` fields.
|
||||
|
||||
```json
|
||||
"/_cluster/stats": {
|
||||
"indices": [
|
||||
"shards.total",
|
||||
"shards.index.shards.min"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Painless triggers
|
||||
|
||||
Painless scripts define triggers for cluster metrics monitors, similar to query or bucket-level monitors that are defined using the extraction query definition option. Painless scripts are comprised of at least one statement and any additional functions you wish to execute.
|
||||
|
||||
The cluster metrics monitor supports up to **ten** triggers.
|
||||
|
||||
In this example, a JSON object creates a trigger that sends an alert when the Cluster Health is yellow. `script` points the `source` to the painless script `ctx.results[0].status == \"yellow\`.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Cluster Health Monitor",
|
||||
"type": "monitor",
|
||||
"monitor_type": "query_level_monitor",
|
||||
"enabled": true,
|
||||
"schedule": {
|
||||
"period": {
|
||||
"unit": "MINUTES",
|
||||
"interval": 1
|
||||
}
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"uri": {
|
||||
"api_type": "CLUSTER_HEALTH",
|
||||
"path": "_cluster/health/",
|
||||
"path_params": "",
|
||||
"url": "http://localhost:9200/_cluster/health/"
|
||||
}
|
||||
}
|
||||
],
|
||||
"triggers": [
|
||||
{
|
||||
"query_level_trigger": {
|
||||
"id": "Tf_L_nwBti6R6Bm-18qC",
|
||||
"name": "Yellow status trigger",
|
||||
"severity": "1",
|
||||
"condition": {
|
||||
"script": {
|
||||
"source": "ctx.results[0].status == \"yellow\"",
|
||||
"lang": "painless"
|
||||
}
|
||||
},
|
||||
"actions": []
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
See [trigger variables](#trigger-variables), for more painless ctx options.
|
||||
|
||||
### Limitations
|
||||
|
||||
Currently, the cluster metrics monitor has the following limitations:
|
||||
|
||||
- You cannot create monitors for remote clusters.
|
||||
- The OpenSearch cluster must be in a state where an index's conditions can be monitored and actions can be executed against the index.
|
||||
- Removing resource permissions from a user will not prevent that user’s preexisting monitors for that resource from executing.
|
||||
- Users with permissions to create monitors are not blocked from creating monitors for resources for which they do not have permissions; however, those monitors will not execute.
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
---
|
||||
layout: default
|
||||
title: Application analytics
|
||||
nav_order: 80
|
||||
---
|
||||
|
||||
# Application analytics
|
||||
|
||||
Application analytics can now create custom Observability Applications to view the availability status of all their systems, where log events can be combined with trace and metric data, into a single view of system health empowering developers and IT Ops to resolve issues faster and with fewer escalations.Application analytics in Observability is where you can select logs, traces, and metrics to be part of an Observability Application that can be monitored for overall health and visualized on one page. This allows you to quickly pivot between logs, traces, and metrics to dig into the source of any issues. Many teams use similar logs and similar traces and need a structured view.
|
||||
|
||||
## Get started with application analytics
|
||||
|
||||
To get started, choose **Observability** in OpenSearch Dashboards, and then choose **Application analytics**.
|
||||
|
||||
### Create an application
|
||||
|
||||
1. Choose **Create application**
|
||||
1. Enter a name for your application and optionally add a description
|
||||
1. Do at least one of the following
|
||||
- Use [PPL]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) to specify the base query.
|
||||
|
||||
You can't change the base query after the application is created.
|
||||
{: .note }
|
||||
|
||||
- Select services & entities from the dropdown or the service map.
|
||||
- Select trace groups from the dropdown or the table.
|
||||
1. Choose **Create**.
|
||||
|
||||
### Create a visualization
|
||||
|
||||
1. Choose the **Log Events** tab
|
||||
1. Use [PPL]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) to build upon your base query
|
||||
1. Move to the **Visualizations** tab to see your visualizations
|
||||
1. Expand the **Save** dropdown menu, enter a name for your visualization, then choose **Save**.
|
||||
|
||||
To see your visualizations, move to the **Panel** tab.
|
||||
|
||||
### Configure availability
|
||||
|
||||
Availability is the status of your application determined by availability levels set on a [time series metric]({{site.url}}{{site.baseurl}}/observability-plugin/app-analytics/#time-series-metric).
|
||||
|
||||
To create an availability level you must configure the following
|
||||
- color: The color of the availability badge on the home page.
|
||||
- name: The text in the availability badge on the home page.
|
||||
- expression: Comparison operator to determine the availability.
|
||||
- value: Value to use when calculating availability.
|
||||
|
||||
|
||||
|
||||
By default, Application analytics shows results from the last 24 hours of your data. To see data from a different timeframe, use the date and time selector.
|
||||
|
||||
#### Time series metric
|
||||
|
||||
A time series metric is any visualization that has a query that spans over a timestamp and is a line chart. Users can use the power of the PPL language to define arbitrary conditions on their logs to create a visualization over time.
|
||||
|
||||
##### Example
|
||||
```
|
||||
source = <index_name> | ... | ... | stats ... by span(<timestamp_field>, 1h)
|
||||
```
|
||||
|
||||
Users can then choose **Line** in visualization configurations to create a time series metric.
|
||||
|
||||
|
||||
|
|
@ -30,4 +30,26 @@ For more information about building PPL queries, see [Piped Processing Language]
|
|||
|
||||
After Dashboards generates a visualization, you must save it if you want to return to it at a later time or if you want to add it to an [operational panel]({{site.url}}{{site.baseurl}}/observability-plugin/operational-panels).
|
||||
|
||||
To save a visualization, expand the save dropdown menu next to **Run**, enter a name for your visualization, then choose **Save**. You can reopen any saved visualizations on the event analytics page.
|
||||
To save a visualization, expand the save dropdown menu next to **Refresh**, enter a name for your visualization, then choose **Save**. You can reopen any saved visualizations on the event analytics page.
|
||||
|
||||
## View logs
|
||||
|
||||
Users have several ways to view their logs.
|
||||
|
||||
### Trace log correlation
|
||||
|
||||
Users who regularly track events across applications have the capability to correlate logs and traces. To view the correlation, users have to index the traces as per the Open Telemetry standards (similar to Trace analytics). Once users add a **TraceId** field to their logs, they can view the trace information that corresponds to the same execution context as the log.
|
||||
|
||||
|
||||
|
||||
### Surrounding events
|
||||
|
||||
Users who want to know more about a log event that they are looking at can select **View surrounding events** to get a bigger picture of what was happening around the time of interest.
|
||||
|
||||
|
||||
|
||||
### Live Tail
|
||||
|
||||
Users watching a live event take place can now configure the interval in which content is refreshed saving the hassle of manually refreshing. This provides users with a faster debugging experience, and allows them to monitor their logs in real-time. Users also have the convenience of choosing the interval at which live tail should update. As this feature mimics the "tail -f" command in command line, it only retrieves the most recent live logs by possibly eliminating a heavy load of live logs.
|
||||
|
||||
|
||||
|
|
@ -151,6 +151,10 @@ search source=accounts | dedup gender consecutive=true | fields account_number,
|
|||
13 | F
|
||||
18 | M
|
||||
|
||||
### Limitations
|
||||
|
||||
The `dedup` command is not rewritten to OpenSearch DSL, it is only executed on the coordination node.
|
||||
|
||||
## eval
|
||||
|
||||
The `eval` command evaluates an expression and appends its result to the search result.
|
||||
|
|
@ -211,6 +215,11 @@ search source=accounts | eval doubleAge = age * 2, ddAge = doubleAge * 2 | field
|
|||
| 28 | 56 | 112
|
||||
| 33 | 66 | 132
|
||||
|
||||
|
||||
### Limitation
|
||||
|
||||
The ``eval`` command is not rewritten to OpenSearch DSL, it is only executed on the coordination node.
|
||||
|
||||
## fields
|
||||
|
||||
Use the `fields` command to keep or remove fields from a search result.
|
||||
|
|
@ -256,6 +265,80 @@ search source=accounts | fields account_number, firstname, lastname | fields - a
|
|||
| Nanette | Bates
|
||||
| Dale | Adams
|
||||
|
||||
|
||||
## parse
|
||||
|
||||
Use the `parse` command to parse a text field using regular expression and append the result to the search result.
|
||||
|
||||
### Syntax
|
||||
|
||||
```sql
|
||||
parse <field> <regular-expression>
|
||||
```
|
||||
|
||||
Field | Description | Required
|
||||
:--- | :--- |:---
|
||||
field | A text field. | Yes
|
||||
regular-expression | The regular expression used to extract new fields from the given test field. If a new field name exists, it will replace the original field. | Yes
|
||||
|
||||
The regular expression is used to match the whole text field of each document with Java regex engine. Each named capture group in the expression will become a new ``STRING`` field.
|
||||
|
||||
*Example 1*: Create new field
|
||||
|
||||
The example shows how to create new field `host` for each document. `host` will be the hostname after `@` in `email` field. Parsing a null field will return an empty string.
|
||||
|
||||
```sql
|
||||
os> source=accounts | parse email '.+@(?<host>.+)' | fields email, host ;
|
||||
fetched rows / total rows = 4/4
|
||||
```
|
||||
|
||||
| email | host
|
||||
:--- | :--- |
|
||||
| amberduke@pyrami.com | pyrami.com
|
||||
| hattiebond@netagy.com | netagy.com
|
||||
| null | null
|
||||
| daleadams@boink.com | boink.com
|
||||
|
||||
*Example 2*: Override the existing field
|
||||
|
||||
The example shows how to override the existing address field with street number removed.
|
||||
|
||||
```sql
|
||||
os> source=accounts | parse address '\d+ (?<address>.+)' | fields address ;
|
||||
fetched rows / total rows = 4/4
|
||||
```
|
||||
|
||||
| address
|
||||
:--- |
|
||||
| Holmes Lane
|
||||
| Bristol Street
|
||||
| Madison Street
|
||||
| Hutchinson Court
|
||||
|
||||
*Example 3*: Filter and sort be casted parsed field
|
||||
|
||||
The example shows how to sort street numbers that are higher than 500 in address field.
|
||||
|
||||
```sql
|
||||
os> source=accounts | parse address '(?<streetNumber>\d+) (?<street>.+)' | where cast(streetNumber as int) > 500 | sort num(streetNumber) | fields streetNumber, street ;
|
||||
fetched rows / total rows = 3/3
|
||||
```
|
||||
|
||||
| streetNumber | street
|
||||
:--- | :--- |
|
||||
| 671 | Bristol Street
|
||||
| 789 | Madison Street
|
||||
| 880 | Holmes Lane
|
||||
|
||||
### Limitations
|
||||
|
||||
A few limitations exist when using the parse command:
|
||||
|
||||
- Fields defined by parse cannot be parsed again. For example, `source=accounts | parse address '\d+ (?<street>.+)' | parse street '\w+ (?<road>\w+)' ;` will fail to return any expressions.
|
||||
- Fields defined by parse cannot be overridden with other commands. For example, when entering `source=accounts | parse address '\d+ (?<street>.+)' | eval street='1' | where street='1' ;` `where` will not match any documents since `street` cannot be overridden.
|
||||
- The text field used by parse cannot be overridden. For example, when entering `source=accounts | parse address '\d+ (?<street>.+)' | eval address='1' ;` `street` will not be parse since address is overridden.
|
||||
- Fields defined by parse cannot be filtered/sorted after using them in the `stats` command. For example, `source=accounts | parse email '.+@(?<host>.+)' | stats avg(age) by host | where host=pyrami.com ;` `where` will not parse the domain listed.
|
||||
|
||||
## rename
|
||||
|
||||
Use the `rename` command to rename one or more fields in the search result.
|
||||
|
|
@ -301,6 +384,10 @@ search source=accounts | rename account_number as an, employer as emp | fields a
|
|||
| 13 | Quility
|
||||
| 18 | null
|
||||
|
||||
### Limitations
|
||||
|
||||
The `rename` command is not rewritten to OpenSearch DSL, it is only executed on the coordination node.
|
||||
|
||||
## sort
|
||||
|
||||
Use the `sort` command to sort search results by a specified field.
|
||||
|
|
@ -547,6 +634,10 @@ search source=accounts | fields firstname, age | head 2;
|
|||
| Amber | 32
|
||||
| Hattie | 36
|
||||
|
||||
### Limitations
|
||||
|
||||
The `head` command is not rewritten to OpenSearch DSL, it is only executed on the coordination node.
|
||||
|
||||
## rare
|
||||
|
||||
Use the `rare` command to find the least common values of all fields in a field list.
|
||||
|
|
@ -590,6 +681,10 @@ search source=accounts | rare age by gender;
|
|||
| M | 32
|
||||
| M | 33
|
||||
|
||||
### Limitations
|
||||
|
||||
The `rare` command is not rewritten to OpenSearch DSL, it is only executed on the coordination node.
|
||||
|
||||
## top {#top-command}
|
||||
|
||||
Use the `top` command to find the most common values of all fields in the field list.
|
||||
|
|
@ -644,6 +739,10 @@ search source=accounts | top 1 age by gender;
|
|||
| F | 28
|
||||
| M | 32
|
||||
|
||||
### Limitations
|
||||
|
||||
The `top` command is not rewritten to OpenSearch DSL, it is only executed on the coordination node.
|
||||
|
||||
## match
|
||||
|
||||
Use the `match` command to search documents that match a `string`, `number`, `date`, or `boolean` value for a given field.
|
||||
|
|
|
|||
|
|
@ -17,3 +17,5 @@ nav_order: 2
|
|||
OpenSearch version | Compatible Java versions | Recommended operating systems
|
||||
:--- | :--- | :---
|
||||
1.x | 8, 11, 15 | Red Hat Enterprise Linux 7, 8; CentOS 7, 8; Amazon Linux 2; Ubuntu 16.04, 18.04, 20.04
|
||||
|
||||
If you plan on running applications besides OpenSearch with different Java requirements, set the `OPENSEARCH_JAVA_HOME` environment variable to the location of the Java installation to be used by OpenSearch. `OPENSEARCH_JAVA_HOME` supersedes the `JAVA_HOME` environment variable.
|
||||
|
|
@ -65,6 +65,24 @@ bin/opensearch-plugin list
|
|||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>1.3.0</td>
|
||||
<td>
|
||||
<pre>opensearch-alerting 1.3.0.0
|
||||
opensearch-anomaly-detection 1.3.0.0
|
||||
opensearch-asynchronous-search 1.3.0.0
|
||||
opensearch-cross-cluster-replication 1.3.0.0
|
||||
opensearch-index-management 1.3.0.0
|
||||
opensearch-job-scheduler 1.3.0.0
|
||||
opensearch-knn 1.3.0.0
|
||||
opensearch-ml 1.3.0.0
|
||||
opensearch-observability 1.3.0.0
|
||||
opensearch-performance-analyzer 1.3.0.0
|
||||
opensearch-reports-scheduler 1.3.0.0
|
||||
opensearch-security 1.3.0.0
|
||||
opensearch-sql 1.3.0.0
|
||||
</pre>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>1.2.4</td>
|
||||
<td>
|
||||
|
|
|
|||
|
|
@ -168,7 +168,7 @@ POST _bulk
|
|||
After indexing into the training index completes, we can call the Train API:
|
||||
|
||||
```json
|
||||
POST /_plugins/_knn/models/_train/my-model
|
||||
POST /_plugins/_knn/models/my-model/_train
|
||||
{
|
||||
"training_index": "train-index",
|
||||
"training_field": "train-field",
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ double | double | DOUBLE
|
|||
keyword | string | VARCHAR
|
||||
text | text | VARCHAR
|
||||
date | timestamp | TIMESTAMP
|
||||
date_nanos | timestamp | TIMESTAMP
|
||||
ip | ip | VARCHAR
|
||||
date | timestamp | TIMESTAMP
|
||||
binary | binary | VARBINARY
|
||||
|
|
@ -54,7 +55,7 @@ The `time` type represents the time of a clock regardless of its timezone. The `
|
|||
|
||||
| Type | Syntax | Range
|
||||
:--- | :--- | :---
|
||||
time | `hh:mm:ss[.fraction]` | `00:00:00.000000` to `23:59:59.999999`
|
||||
time | `hh:mm:ss[.fraction]` | `00:00:00.0000000000` to `23:59:59.9999999999`
|
||||
|
||||
### Datetime
|
||||
|
||||
|
|
@ -62,7 +63,7 @@ The `datetime` type is a combination of date and time. It doesn't contain timezo
|
|||
|
||||
| Type | Syntax | Range
|
||||
:--- | :--- | :---
|
||||
datetime | `yyyy-MM-dd hh:mm:ss[.fraction]` | `0001-01-01 00:00:00.000000` to `9999-12-31 23:59:59.999999`
|
||||
datetime | `yyyy-MM-dd hh:mm:ss[.fraction]` | `0001-01-01 00:00:00.0000000000` to `9999-12-31 23:59:59.9999999999`
|
||||
|
||||
### Timestamp
|
||||
|
||||
|
|
@ -72,7 +73,7 @@ The `timestamp` type is stored differently from the other types. It's converted
|
|||
|
||||
| Type | Syntax | Range
|
||||
:--- | :--- | :---
|
||||
timestamp | `yyyy-MM-dd hh:mm:ss[.fraction]` | `0001-01-01 00:00:01.000000` UTC to `9999-12-31 23:59:59.999999`
|
||||
timestamp | `yyyy-MM-dd hh:mm:ss[.fraction]` | `0001-01-01 00:00:01.9999999999` UTC to `9999-12-31 23:59:59.9999999999`
|
||||
|
||||
### Interval
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 2.6 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 6.7 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.4 MiB |
|
|
@ -9,6 +9,7 @@ permalink: /version-history/
|
|||
|
||||
OpenSearch version | Release highlights | Release date
|
||||
:--- | :--- | :--- | :---
|
||||
[1.3.0]() | Adds Model Type Validation to Validate Detector API, continuous transforms, custom actions, applied policy parameter to Explain API, default action retries, and new rollover and transition conditions to Index Management, new ML Commons plugin, parse command to SQL, Application Analytics, Live Tail, Correlation, and Events Flyout to Observbility, and auto backport and support for OPENSEARCH_JAVA_HOME to Performance Analyzer. Bug fixes. | 17 March 2022
|
||||
[1.2.4](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.2.4.md) | Updates Performance Analyzer, SQL, and Security plugins to Log4j 2.17.1, Alerting and Job Scheduler to cron-utils 9.1.6, and gson in Anomaly Detection and SQL. | 18 January 2022
|
||||
[1.2.3](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.2.3.md) | Updates the version of Log4j used in OpenSearch to Log4j 2.17.0 as recommended by the advisory in [CVE-2021-45105](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105). | 22 December 2021
|
||||
[1.2.0](https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.2.0.md) | Adds observability, new validation API for Anomaly Detection, shard-level indexing back-pressure, new "match" query type for SQL and PPL, support for Faiss libraries in k-NN, and custom Dashboards branding. | 23 November 2021
|
||||
|
|
|
|||
Loading…
Reference in New Issue