Merge pull request #37 from opensearch-project/index-transforms
Added new section on transform jobs
This commit is contained in:
Keith Chan
GitHub
commit
76bed73dd3
|
|
@ -0,0 +1,156 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: Index transforms
|
||||||
|
nav_order: 40
|
||||||
|
parent: Index management
|
||||||
|
has_children: true
|
||||||
|
has_toc: false
|
||||||
|
---
|
||||||
|
|
||||||
|
# Index transforms
|
||||||
|
|
||||||
|
Whereas index rollup jobs let you reduce data granularity by rolling up old data into condensed indices, transform jobs let you create a different, summarized view of your data centered around certain fields, so you can visualize or analyze the data in different ways.
|
||||||
|
|
||||||
|
For example, suppose that you have airline data that’s scattered across multiple fields and categories, and you want to view a summary of the data that’s organized by airline, quarter, and then price. You can use a transform job to create a new, summarized index that’s organized by those specific categories.
|
||||||
|
|
||||||
|
You can use transform jobs in two ways:
|
||||||
|
|
||||||
|
1. Use the OpenSearch Dashboards UI to specify the index you want to transform and any optional data filters you want to use to filter the original index. Then select the fields you want to transform and the aggregations to use in the transformation. Finally, define a schedule for your job to follow.
|
||||||
|
2. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow.
|
||||||
|
|
||||||
|
OpenSearch Dashboards provides a detailed summary of the jobs you created and their relevant information, such as associated indices and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards.
|
||||||
|
|
||||||
|
Your use cases will help you decide which method to use to create transform jobs.
|
||||||
|
|
||||||
|
## Create a transform job
|
||||||
|
|
||||||
|
If you don't have any data in your cluster, you can use the sample flight data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose **Index Management**. Select **Transform Jobs**, and choose **Create Transform Job**.
|
||||||
|
|
||||||
|
### Step 1: Choose indices
|
||||||
|
|
||||||
|
1. In the **Job name and description** section, specify a name and an optional description for your job.
|
||||||
|
2. In the **Indices** section, select the source and target index. You can either select an existing target index or create a new one by entering a name for your new index. If you want to transform just a subset of your source index, choose **Add Data Filter**, and use the OpenSearch query DSL to specify a subset of your source index. For more information about the OpenSearch query DSL, see [query DSL](../../opensearch/query-dsl/).
|
||||||
|
3. Choose **Next**.
|
||||||
|
|
||||||
|
### Step 2: Select fields to transform
|
||||||
|
|
||||||
|
After specifying the indices, you can select the fields you want to use in your transform job, as well as whether to use groupings or aggregations.
|
||||||
|
|
||||||
|
You can use groupings to place your data into separate buckets in your transformed index. For example, if you want to group all of the airport destinations within the sample flight data, you can group the `DestAirportID` field into a target field of `DestAirportID_terms` field, and you can find the grouped airport IDs in your transformed index after the transform job finishes.
|
||||||
|
|
||||||
|
On the other hand, aggregations let you perform simple calculations. For example, you can include an aggregation in your transform job to define a new field of `sum_of_total_ticket_price` that calculates the sum of all airplane tickets, and then analyze the newly summer data within your transformed index.
|
||||||
|
|
||||||
|
1. In the data table, select the fields you want to transform and expand the drop-down menu within the column header to choose the grouping or aggregation you want to use.
|
||||||
|
|
||||||
|
Currently, transform jobs support histogram, date_histogram, and terms groupings. For more information about groupings, see [Bucket Aggregations](../../opensearch/bucket-agg/). In terms of aggregations, you can select from `sum`, `avg`, `max`, `min`, `value_count`, `percentiles`, and `scripted_metric`. For more information about aggregations, see [Metric Aggregations](../../opensearch/metric-agg/).
|
||||||
|
|
||||||
|
2. Repeat step 1 for any other fields that you want to transform.
|
||||||
|
3. After selecting the fields that you want to transform and verifying the transformation, choose **Next**.
|
||||||
|
|
||||||
|
### Step 3: Specify a schedule
|
||||||
|
|
||||||
|
You can configure transform jobs to run once or multiple times on a schedule. Transform jobs are enabled by default.
|
||||||
|
|
||||||
|
1. For **transformation execution frequency**, select either **Define by fixed interval** and specify a **transform interval**.
|
||||||
|
2. Under **Advanced**, specify an optional amount for **Pages per execution**. A larger number means more data is processed in each search request, but also uses more memory and causes higher latency. Exceeding allowed memory limits can cause exceptions and errors to occur.
|
||||||
|
3. Choose **Next**.
|
||||||
|
|
||||||
|
### Step 4: Review and confirm details
|
||||||
|
|
||||||
|
After confirming your transform job’s details are correct, choose **Create Transform Job**. If you want to edit any part of the job, choose **Edit** of the section you want to change, and make the necessary changes. You can’t change aggregations or groupings after creating a job.
|
||||||
|
|
||||||
|
### Step 5: Search through the transformed index.
|
||||||
|
|
||||||
|
Once the transform job finishes, you can use the `_search` API operation to search the target index.
|
||||||
|
|
||||||
|
```json
|
||||||
|
GET <target_index>/_search
|
||||||
|
```
|
||||||
|
|
||||||
|
For example, after running a transform job that transforms the flight data based on a `DestAirportID` field, you can run the following request that returns all of the fields that have a value of `SFO`.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
GET finished_flight_job/_search
|
||||||
|
{
|
||||||
|
"query": {
|
||||||
|
"match": {
|
||||||
|
"DestAirportID_terms" : "SFO"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"took" : 3,
|
||||||
|
"timed_out" : false,
|
||||||
|
"_shards" : {
|
||||||
|
"total" : 5,
|
||||||
|
"successful" : 5,
|
||||||
|
"skipped" : 0,
|
||||||
|
"failed" : 0
|
||||||
|
},
|
||||||
|
"hits" : {
|
||||||
|
"total" : {
|
||||||
|
"value" : 4,
|
||||||
|
"relation" : "eq"
|
||||||
|
},
|
||||||
|
"max_score" : 3.845883,
|
||||||
|
"hits" : [
|
||||||
|
{
|
||||||
|
"_index" : "finished_flight_job",
|
||||||
|
"_type" : "_doc",
|
||||||
|
"_id" : "dSNKGb8U3OJOmC4RqVCi1Q",
|
||||||
|
"_score" : 3.845883,
|
||||||
|
"_source" : {
|
||||||
|
"transform._id" : "sample_flight_job",
|
||||||
|
"transform._doc_count" : 14,
|
||||||
|
"Carrier_terms" : "Kibana Airlines",
|
||||||
|
"DestAirportID_terms" : "SFO"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"_index" : "finished_flight_job",
|
||||||
|
"_type" : "_doc",
|
||||||
|
"_id" : "_D7oqOy7drx9E-MG96U5RA",
|
||||||
|
"_score" : 3.845883,
|
||||||
|
"_source" : {
|
||||||
|
"transform._id" : "sample_flight_job",
|
||||||
|
"transform._doc_count" : 14,
|
||||||
|
"Carrier_terms" : "Logstash Airways",
|
||||||
|
"DestAirportID_terms" : "SFO"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"_index" : "finished_flight_job",
|
||||||
|
"_type" : "_doc",
|
||||||
|
"_id" : "YuZ8tOt1OsBA54e84WuAEw",
|
||||||
|
"_score" : 3.6988301,
|
||||||
|
"_source" : {
|
||||||
|
"transform._id" : "sample_flight_job",
|
||||||
|
"transform._doc_count" : 11,
|
||||||
|
"Carrier_terms" : "ES-Air",
|
||||||
|
"DestAirportID_terms" : "SFO"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"_index" : "finished_flight_job",
|
||||||
|
"_type" : "_doc",
|
||||||
|
"_id" : "W_-e7bVmH6eu8veJeK8ZxQ",
|
||||||
|
"_score" : 3.6988301,
|
||||||
|
"_source" : {
|
||||||
|
"transform._id" : "sample_flight_job",
|
||||||
|
"transform._doc_count" : 10,
|
||||||
|
"Carrier_terms" : "JetBeats",
|
||||||
|
"DestAirportID_terms" : "SFO"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,714 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: Transforms APIs
|
||||||
|
nav_order: 45
|
||||||
|
parent: Index Transforms
|
||||||
|
grand_parent: Index management
|
||||||
|
has_toc: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Transforms APIs
|
||||||
|
|
||||||
|
Aside from using OpenSearch Dashboards, you can also use the REST API to create, start, stop, and complete other operations relative to transform jobs.
|
||||||
|
|
||||||
|
#### Table of contents
|
||||||
|
- TOC
|
||||||
|
{:toc}
|
||||||
|
|
||||||
|
## Create a transform job
|
||||||
|
|
||||||
|
Creates a transform job.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
PUT _opendistro/_transform/<transform_id>
|
||||||
|
|
||||||
|
{
|
||||||
|
"transform": {
|
||||||
|
"enabled": true,
|
||||||
|
"schedule": {
|
||||||
|
"interval": {
|
||||||
|
"period": 1,
|
||||||
|
"unit": "Minutes",
|
||||||
|
"start_time": 1602100553
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"description": "Sample transform job",
|
||||||
|
"source_index": "sample_index",
|
||||||
|
"target_index": "sample_target",
|
||||||
|
"data_selection_query": {
|
||||||
|
"match_all": {}
|
||||||
|
},
|
||||||
|
"page_size": 1,
|
||||||
|
"groups": [
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "customer_gender",
|
||||||
|
"target_field": "gender"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "day_of_week",
|
||||||
|
"target_field": "day"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"aggregations": {
|
||||||
|
"quantity": {
|
||||||
|
"sum": {
|
||||||
|
"field": "total_quantity"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_id": "sample",
|
||||||
|
"_version": 7,
|
||||||
|
"_seq_no": 13,
|
||||||
|
"_primary_term": 1,
|
||||||
|
"transform": {
|
||||||
|
"transform_id": "sample",
|
||||||
|
"schema_version": 7,
|
||||||
|
"schedule": {
|
||||||
|
"interval": {
|
||||||
|
"start_time": 1621467964243,
|
||||||
|
"period": 1,
|
||||||
|
"unit": "Minutes"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"metadata_id": null,
|
||||||
|
"updated_at": 1621467964243,
|
||||||
|
"enabled": true,
|
||||||
|
"enabled_at": 1621467964243,
|
||||||
|
"description": "Sample transform job",
|
||||||
|
"source_index": "sample_index",
|
||||||
|
"data_selection_query": {
|
||||||
|
"match_all": {
|
||||||
|
"boost": 1.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"target_index": "sample_target",
|
||||||
|
"roles": [],
|
||||||
|
"page_size": 1,
|
||||||
|
"groups": [
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "customer_gender",
|
||||||
|
"target_field": "gender"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "day_of_week",
|
||||||
|
"target_field": "day"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"aggregations": {
|
||||||
|
"quantity": {
|
||||||
|
"sum": {
|
||||||
|
"field": "total_quantity"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
You can specify the following options in the HTTP request body:
|
||||||
|
|
||||||
|
Option | Data Type | Description | Required
|
||||||
|
:--- | :--- | :--- | :---
|
||||||
|
enabled | Boolean | If true, the transform job is enabled at creation. | No
|
||||||
|
schedule | JSON | The schedule the transform job runs on. | Yes
|
||||||
|
start_time | Integer | The Unix epoch time of the transform job's start time. | Yes
|
||||||
|
description | String | Describes the transform job. | No
|
||||||
|
metadata_id | String | Any metadata to be associated with the transform job. | No
|
||||||
|
source_index | String | The source index whose data to transform. | Yes
|
||||||
|
target_index | String | The target index the newly transformed data is added into. You can create a new index or update an existing one. | Yes
|
||||||
|
data_selection_query | JSON | The query DSL to use to filter a subset of the source index for the transform job. See [query DSL](../../../opensearch/query-dsl) for more information. | Yes
|
||||||
|
page_size | Integer | The number of fields to transform at a time. Higher number means higher performance but requires more memory and can cause higher latency. (Default: 1) | Yes
|
||||||
|
groups | Array | Specifies the grouping(s) to use in the transform job. Supported groups are `terms`, `histogram`, and `date_histogram`. For more information, see [Bucket Aggregations](../../../opensearch/bucket-agg). | Yes if not using aggregations
|
||||||
|
source_field | String | The field(s) to transform | Yes
|
||||||
|
aggregations | JSON | The aggregations to use in the transform job. Supported aggregations are: `sum`, `max`, `min`, `value_count`, `avg`, `scripted_metric`, and `percentiles`. For more information, see [Metric Aggregations](../../../opensearch/metric-agg). | Yes if not using groups
|
||||||
|
|
||||||
|
## Update a transform job
|
||||||
|
|
||||||
|
Updates a transform job if `transform_id` already exists.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
PUT _opendistro/_transform/<transform_id>
|
||||||
|
|
||||||
|
{
|
||||||
|
"transform": {
|
||||||
|
"enabled": true,
|
||||||
|
"schedule": {
|
||||||
|
"interval": {
|
||||||
|
"period": 1,
|
||||||
|
"unit": "Minutes",
|
||||||
|
"start_time": 1602100553
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"description": "Sample transform job",
|
||||||
|
"source_index": "sample_index",
|
||||||
|
"target_index": "sample_target",
|
||||||
|
"data_selection_query": {
|
||||||
|
"match_all": {}
|
||||||
|
},
|
||||||
|
"page_size": 1,
|
||||||
|
"groups": [
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "customer_gender",
|
||||||
|
"target_field": "gender"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "day_of_week",
|
||||||
|
"target_field": "day"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"aggregations": {
|
||||||
|
"quantity": {
|
||||||
|
"sum": {
|
||||||
|
"field": "total_quantity"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_id": "sample",
|
||||||
|
"_version": 2,
|
||||||
|
"_seq_no": 14,
|
||||||
|
"_primary_term": 1,
|
||||||
|
"transform": {
|
||||||
|
"transform_id": "sample",
|
||||||
|
"schema_version": 7,
|
||||||
|
"schedule": {
|
||||||
|
"interval": {
|
||||||
|
"start_time": 1602100553,
|
||||||
|
"period": 1,
|
||||||
|
"unit": "Minutes"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"metadata_id": null,
|
||||||
|
"updated_at": 1621889843874,
|
||||||
|
"enabled": true,
|
||||||
|
"enabled_at": 1621889843874,
|
||||||
|
"description": "Sample transform job",
|
||||||
|
"source_index": "sample_index",
|
||||||
|
"data_selection_query": {
|
||||||
|
"match_all": {
|
||||||
|
"boost": 1.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"target_index": "sample_target",
|
||||||
|
"roles": [],
|
||||||
|
"page_size": 1,
|
||||||
|
"groups": [
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "customer_gender",
|
||||||
|
"target_field": "gender"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "day_of_week",
|
||||||
|
"target_field": "day"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"aggregations": {
|
||||||
|
"quantity": {
|
||||||
|
"sum": {
|
||||||
|
"field": "total_quantity"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The `Update` operation supports the following URL parameters:
|
||||||
|
|
||||||
|
Parameter | Description | Required
|
||||||
|
:---| :--- | :---
|
||||||
|
`if_seq_no` | Only perform the transform operation if the last operation that changed the transform job has the specified sequence number. | No
|
||||||
|
`if_primary_term` | Only perform the transform operation if the last operation that changed the transform job has the specified sequence term. | No
|
||||||
|
|
||||||
|
## Get a transform job's details
|
||||||
|
|
||||||
|
Returns a transform job's details.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
GET _opendistro/_transform/<transform_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_id": "sample",
|
||||||
|
"_version": 7,
|
||||||
|
"_seq_no": 13,
|
||||||
|
"_primary_term": 1,
|
||||||
|
"transform": {
|
||||||
|
"transform_id": "sample",
|
||||||
|
"schema_version": 7,
|
||||||
|
"schedule": {
|
||||||
|
"interval": {
|
||||||
|
"start_time": 1621467964243,
|
||||||
|
"period": 1,
|
||||||
|
"unit": "Minutes"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"metadata_id": null,
|
||||||
|
"updated_at": 1621467964243,
|
||||||
|
"enabled": true,
|
||||||
|
"enabled_at": 1621467964243,
|
||||||
|
"description": "Sample transform job",
|
||||||
|
"source_index": "sample_index",
|
||||||
|
"data_selection_query": {
|
||||||
|
"match_all": {
|
||||||
|
"boost": 1.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"target_index": "sample_target",
|
||||||
|
"roles": [],
|
||||||
|
"page_size": 1,
|
||||||
|
"groups": [
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "customer_gender",
|
||||||
|
"target_field": "gender"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "day_of_week",
|
||||||
|
"target_field": "day"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"aggregations": {
|
||||||
|
"quantity": {
|
||||||
|
"sum": {
|
||||||
|
"field": "total_quantity"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
You can also get details of all transform jobs by omitting `transform_id`.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
GET _opendistro/_transform/
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"total_transforms": 1,
|
||||||
|
"transforms": [
|
||||||
|
{
|
||||||
|
"_id": "sample",
|
||||||
|
"_seq_no": 13,
|
||||||
|
"_primary_term": 1,
|
||||||
|
"transform": {
|
||||||
|
"transform_id": "sample",
|
||||||
|
"schema_version": 7,
|
||||||
|
"schedule": {
|
||||||
|
"interval": {
|
||||||
|
"start_time": 1621467964243,
|
||||||
|
"period": 1,
|
||||||
|
"unit": "Minutes"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"metadata_id": null,
|
||||||
|
"updated_at": 1621467964243,
|
||||||
|
"enabled": true,
|
||||||
|
"enabled_at": 1621467964243,
|
||||||
|
"description": "Sample transform job",
|
||||||
|
"source_index": "sample_index",
|
||||||
|
"data_selection_query": {
|
||||||
|
"match_all": {
|
||||||
|
"boost": 1.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"target_index": "sample_target",
|
||||||
|
"roles": [],
|
||||||
|
"page_size": 1,
|
||||||
|
"groups": [
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "customer_gender",
|
||||||
|
"target_field": "gender"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "day_of_week",
|
||||||
|
"target_field": "day"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"aggregations": {
|
||||||
|
"quantity": {
|
||||||
|
"sum": {
|
||||||
|
"field": "total_quantity"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
You can specify these options as the `GET` API operation’s URL parameters to filter results:
|
||||||
|
|
||||||
|
Parameter | Description | Required
|
||||||
|
:--- | :--- | :---
|
||||||
|
from | The starting index to search from. (Default: 0) | No
|
||||||
|
size | Specifies the amount of results to return (Default: 10) | No
|
||||||
|
search |The search term to use to filter results. | No
|
||||||
|
sortField | The field to sort results with. | No
|
||||||
|
sortDirection | Specifies the direction to sort results in. Can be `ASC` or `DESC`. (Default: ASC) | No
|
||||||
|
|
||||||
|
For example, this request returns two results starting from the eighth index.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
GET /_opendistro/_transform/?size=2&from=8
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"total_transforms": 18,
|
||||||
|
"transforms": [
|
||||||
|
{
|
||||||
|
"_id": "sample8",
|
||||||
|
"_seq_no": 93,
|
||||||
|
"_primary_term": 1,
|
||||||
|
"transform": {
|
||||||
|
"transform_id": "sample8",
|
||||||
|
"schema_version": 7,
|
||||||
|
"schedule": {
|
||||||
|
"interval": {
|
||||||
|
"start_time": 1622063596812,
|
||||||
|
"period": 1,
|
||||||
|
"unit": "Minutes"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"metadata_id": "y4hFAB2ZURQ2dzY7BAMxWA",
|
||||||
|
"updated_at": 1622063657233,
|
||||||
|
"enabled": false,
|
||||||
|
"enabled_at": null,
|
||||||
|
"description": "Sample transform job",
|
||||||
|
"source_index": "sample_index3",
|
||||||
|
"data_selection_query": {
|
||||||
|
"match_all": {
|
||||||
|
"boost": 1.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"target_index": "sample_target3",
|
||||||
|
"roles": [],
|
||||||
|
"page_size": 1,
|
||||||
|
"groups": [
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "customer_gender",
|
||||||
|
"target_field": "gender"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "day_of_week",
|
||||||
|
"target_field": "day"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"aggregations": {
|
||||||
|
"quantity": {
|
||||||
|
"sum": {
|
||||||
|
"field": "total_quantity"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"_id": "sample9",
|
||||||
|
"_seq_no": 98,
|
||||||
|
"_primary_term": 1,
|
||||||
|
"transform": {
|
||||||
|
"transform_id": "sample9",
|
||||||
|
"schema_version": 7,
|
||||||
|
"schedule": {
|
||||||
|
"interval": {
|
||||||
|
"start_time": 1622063598065,
|
||||||
|
"period": 1,
|
||||||
|
"unit": "Minutes"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"metadata_id": "x8tCIiYMTE3veSbIJkit5A",
|
||||||
|
"updated_at": 1622063658388,
|
||||||
|
"enabled": false,
|
||||||
|
"enabled_at": null,
|
||||||
|
"description": "Sample transform job",
|
||||||
|
"source_index": "sample_index4",
|
||||||
|
"data_selection_query": {
|
||||||
|
"match_all": {
|
||||||
|
"boost": 1.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"target_index": "sample_target4",
|
||||||
|
"roles": [],
|
||||||
|
"page_size": 1,
|
||||||
|
"groups": [
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "customer_gender",
|
||||||
|
"target_field": "gender"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "day_of_week",
|
||||||
|
"target_field": "day"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"aggregations": {
|
||||||
|
"quantity": {
|
||||||
|
"sum": {
|
||||||
|
"field": "total_quantity"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Start a transform job
|
||||||
|
|
||||||
|
Transform jobs created using the API are automatically enabled, but if you ever need to enable a job, you can use the `start` API operation.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
POST _opendistro/<transform_id>/_start
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"acknowledged": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stop a transform job
|
||||||
|
|
||||||
|
Stops/disables a transform job.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
POST _opendistro/<transform_id>/_stop
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"acknowledged": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Get the status of a transform job
|
||||||
|
|
||||||
|
Returns the status and metadata of a transform job.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
GET _opendistro/<transform_id>/_explain
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"sample": {
|
||||||
|
"metadata_id": "PzmjweME5xbgkenl9UpsYw",
|
||||||
|
"transform_metadata": {
|
||||||
|
"transform_id": "sample",
|
||||||
|
"last_updated_at": 1621883525873,
|
||||||
|
"status": "finished",
|
||||||
|
"failure_reason": "null",
|
||||||
|
"stats": {
|
||||||
|
"pages_processed": 0,
|
||||||
|
"documents_processed": 0,
|
||||||
|
"documents_indexed": 0,
|
||||||
|
"index_time_in_millis": 0,
|
||||||
|
"search_time_in_millis": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Preview a transform job's results
|
||||||
|
|
||||||
|
Returns a preview of what a transformed index would look like.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
POST _opendistro/_transform/_preview
|
||||||
|
|
||||||
|
{
|
||||||
|
"transform": {
|
||||||
|
"enabled": false,
|
||||||
|
"schedule": {
|
||||||
|
"interval": {
|
||||||
|
"period": 1,
|
||||||
|
"unit": "Minutes",
|
||||||
|
"start_time": 1602100553
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"description": "test transform",
|
||||||
|
"source_index": "sample_index",
|
||||||
|
"target_index": "sample_target",
|
||||||
|
"data_selection_query": {
|
||||||
|
"match_all": {}
|
||||||
|
},
|
||||||
|
"page_size": 10,
|
||||||
|
"groups": [
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "customer_gender",
|
||||||
|
"target_field": "gender"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"terms": {
|
||||||
|
"source_field": "day_of_week",
|
||||||
|
"target_field": "day"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"aggregations": {
|
||||||
|
"quantity": {
|
||||||
|
"sum": {
|
||||||
|
"field": "total_quantity"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"documents" : [
|
||||||
|
{
|
||||||
|
"quantity" : 862.0,
|
||||||
|
"gender" : "FEMALE",
|
||||||
|
"day" : "Friday"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"quantity" : 682.0,
|
||||||
|
"gender" : "FEMALE",
|
||||||
|
"day" : "Monday"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"quantity" : 772.0,
|
||||||
|
"gender" : "FEMALE",
|
||||||
|
"day" : "Saturday"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"quantity" : 669.0,
|
||||||
|
"gender" : "FEMALE",
|
||||||
|
"day" : "Sunday"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"quantity" : 887.0,
|
||||||
|
"gender" : "FEMALE",
|
||||||
|
"day" : "Thursday"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Delete a transform job
|
||||||
|
|
||||||
|
Deletes a transform job. This operation does not delete the source or target indices.
|
||||||
|
|
||||||
|
**Sample Request**
|
||||||
|
|
||||||
|
```json
|
||||||
|
DELETE _opendistro/_transform/<transform_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sample Response**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"took": 205,
|
||||||
|
"errors": false,
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"delete": {
|
||||||
|
"_index": ".opendistro-ism-config",
|
||||||
|
"_type": "_doc",
|
||||||
|
"_id": "sample",
|
||||||
|
"_version": 4,
|
||||||
|
"result": "deleted",
|
||||||
|
"forced_refresh": true,
|
||||||
|
"_shards": {
|
||||||
|
"total": 2,
|
||||||
|
"successful": 1,
|
||||||
|
"failed": 0
|
||||||
|
},
|
||||||
|
"_seq_no": 6,
|
||||||
|
"_primary_term": 1,
|
||||||
|
"status": 200
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
---
|
---
|
||||||
layout: default
|
layout: default
|
||||||
title: Refresh Search Analyzer
|
title: Refresh Search Analyzer
|
||||||
nav_order: 40
|
nav_order: 50
|
||||||
parent: Index management
|
parent: Index management
|
||||||
has_children: false
|
has_children: false
|
||||||
redirect_from: /docs/ism/refresh-analyzer/
|
redirect_from: /docs/ism/refresh-analyzer/
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue