Merge pull request #447 from opensearch-project/ism-show-policy

Added show policy to ISM explain
This commit is contained in:
Naarcha-AWS 2022-03-17 13:26:43 -05:00 committed by GitHub
commit 3eec39a4e2
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 42 additions and 6 deletions

View File

@ -9,7 +9,7 @@ has_toc: false
# Index transforms # Index transforms
Whereas index rollup jobs let you reduce data granularity by rolling up old data into condensed indices, transform jobs let you create a different, summarized view of your data centered around certain fields, so you can visualize or analyze the data in different ways. Whereas index rollup jobs let you reduce data granularity by rolling up old data into condensed indexes, transform jobs let you create a different, summarized view of your data centered around certain fields, so you can visualize or analyze the data in different ways.
For example, suppose that you have airline data thats scattered across multiple fields and categories, and you want to view a summary of the data thats organized by airline, quarter, and then price. You can use a transform job to create a new, summarized index thats organized by those specific categories. For example, suppose that you have airline data thats scattered across multiple fields and categories, and you want to view a summary of the data thats organized by airline, quarter, and then price. You can use a transform job to create a new, summarized index thats organized by those specific categories.
@ -18,7 +18,7 @@ You can use transform jobs in two ways:
1. Use the OpenSearch Dashboards UI to specify the index you want to transform and any optional data filters you want to use to filter the original index. Then select the fields you want to transform and the aggregations to use in the transformation. Finally, define a schedule for your job to follow. 1. Use the OpenSearch Dashboards UI to specify the index you want to transform and any optional data filters you want to use to filter the original index. Then select the fields you want to transform and the aggregations to use in the transformation. Finally, define a schedule for your job to follow.
1. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow. 1. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow.
OpenSearch Dashboards provides a detailed summary of the jobs you created and their relevant information, such as associated indices and job statuses. You can review and edit your jobs details and selections before creation, and even preview a transformed indexs data as youre choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards. OpenSearch Dashboards provides a detailed summary of the jobs you created and their relevant information, such as associated indexes and job statuses. You can review and edit your jobs details and selections before creation, and even preview a transformed indexs data as youre choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards.
Your use cases will help you decide which method to use to create transform jobs. Your use cases will help you decide which method to use to create transform jobs.
@ -26,7 +26,7 @@ Your use cases will help you decide which method to use to create transform jobs
If you don't have any data in your cluster, you can use the sample flight data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose **Index Management**. Select **Transform Jobs**, and choose **Create Transform Job**. If you don't have any data in your cluster, you can use the sample flight data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose **Index Management**. Select **Transform Jobs**, and choose **Create Transform Job**.
### Step 1: Choose indices ### Step 1: Choose indexes
1. In the **Job name and description** section, specify a name and an optional description for your job. 1. In the **Job name and description** section, specify a name and an optional description for your job.
2. In the **Indices** section, select the source and target index. You can either select an existing target index or create a new one by entering a name for your new index. If you want to transform just a subset of your source index, choose **Edit data filter**, and use the OpenSearch query DSL to specify a subset of your source index. For more information about the OpenSearch query DSL, see [query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/). 2. In the **Indices** section, select the source and target index. You can either select an existing target index or create a new one by entering a name for your new index. If you want to transform just a subset of your source index, choose **Edit data filter**, and use the OpenSearch query DSL to specify a subset of your source index. For more information about the OpenSearch query DSL, see [query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/).
@ -34,7 +34,7 @@ If you don't have any data in your cluster, you can use the sample flight data w
### Step 2: Select fields to transform ### Step 2: Select fields to transform
After specifying the indices, you can select the fields you want to use in your transform job, as well as whether to use groupings or aggregations. After specifying the indexes, you can select the fields you want to use in your transform job, as well as whether to use groupings or aggregations.
You can use groupings to place your data into separate buckets in your transformed index. For example, if you want to group all of the airport destinations within the sample flight data, you can group the `DestAirportID` field into a target field of `DestAirportID_terms` field, and you can find the grouped airport IDs in your transformed index after the transform job finishes. You can use groupings to place your data into separate buckets in your transformed index. For example, if you want to group all of the airport destinations within the sample flight data, you can group the `DestAirportID` field into a target field of `DestAirportID_terms` field, and you can find the grouped airport IDs in your transformed index after the transform job finishes.

View File

@ -713,7 +713,7 @@ POST _plugins/_transform/_preview
Introduced 1.0 Introduced 1.0
{: .label .label-purple } {: .label .label-purple }
Deletes a transform job. This operation does not delete the source or target indices. Deletes a transform job. This operation does not delete the source or target indexes.
**Sample Request** **Sample Request**

View File

@ -176,6 +176,8 @@ Introduced 1.0
Updates a policy. Use the `seq_no` and `primary_term` parameters to update an existing policy. If these numbers don't match the existing policy or the policy doesn't exist, ISM throws an error. Updates a policy. Use the `seq_no` and `primary_term` parameters to update an existing policy. If these numbers don't match the existing policy or the policy doesn't exist, ISM throws an error.
It's possible that the policy currently applied to your index isn't the most up-to-date policy available. To see what policy is currently applied to your index, see [Explain index]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#explain-index). To get the most up-to-date version of a policy, see [Get policy]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#get-policy).
#### Request #### Request
```json ```json
@ -475,6 +477,40 @@ GET _plugins/_ism/explain/index_1
} }
``` ```
Optionally, you can add the `show_policy` parameter to your request's path to get the policy that is currently applied to your index, which is useful for seeing whether the policy applied to your index is the latest one. To get the most up-to-date policy, see [Get Policy API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#get-policy).
#### Request
```json
GET _plugins/_ism/explain/index_1?show_policy=true
```
#### Sample response
```json
{
"index_1": {
"index.plugins.index_state_management.policy_id": "sample-policy",
"index.opendistro.index_state_management.policy_id": "sample-policy",
"index": "index_1",
"index_uuid": "gCFlS_zcTdih8xyxf3jQ-A",
"policy_id": "sample-policy",
"enabled": true,
"policy": {
"policy_id": "sample-policy",
"description": "ingesting logs",
"last_updated_time": 1647284980148,
"schema_version": 13,
"error_notification": null,
"default_state": "ingest",
"states": [...],
"ism_template": null
}
},
"total_managed_indices": 1
}
```
The `plugins.index_state_management.policy_id` setting is deprecated starting from ODFE version 1.13.0. We retain this field in the response API for consistency. The `plugins.index_state_management.policy_id` setting is deprecated starting from ODFE version 1.13.0. We retain this field in the response API for consistency.
--- ---

View File

@ -168,7 +168,7 @@ POST _bulk
After indexing into the training index completes, we can call the Train API: After indexing into the training index completes, we can call the Train API:
```json ```json
POST /_plugins/_knn/models/_train/my-model POST /_plugins/_knn/models/my-model/_train
{ {
"training_index": "train-index", "training_index": "train-index",
"training_field": "train-field", "training_field": "train-field",