opensearch-docs-cn/_data-prepper/pipelines/configuration/processors/translate.md

177 lines
6.7 KiB
Markdown
Raw Permalink Normal View History

Add documentation for Data Prepper translate processor (#5122) * Add translate processor docs Signed-off-by: Hai Yan <oeyh@amazon.com> * Fix a table display issue Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments: tweak sentences; add type info; add top level options Signed-off-by: Hai Yan <oeyh@amazon.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/translate.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update translate.md * Update translate.md * Update _data-prepper/pipelines/configuration/processors/translate.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-10-12 09:53:20 -04:00
---
layout: default
title: translate
parent: Processors
grand_parent: Pipelines
nav_order: 117
---
# translate
The `translate` processor transforms values in events into preconfigured values.
## Basic usage
To use the `translate` processor, create the following `pipeline.yaml` file:
```yaml
translate-pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- translate:
mappings:
- source: "status"
targets:
- target: "translated_result"
map:
404: "Not Found"
sink:
- stdout:
```
Then create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` file with the path of a file containing the following JSON data:
```json
{ "status": "404" }
```
The `translate` processor configuration in `pipeline.yaml` retrieves the `source` value from the event data and compares it against the keys specified under the `targets`.
When a match is found, the processor places the corresponding mapped value into the `target` key provided in the configuration.
When you run Data Prepper with the previous `pipeline.yaml` file, you should receive the following output:
```json
{
"status": "404",
"translated_result": "Not Found"
}
```
## Advanced options
The following example shows a more involved mapping with additional configurations for the `translate` processor:
```yaml
processor:
- translate:
mappings:
- source: "status"
targets:
- target: "translated_result"
map:
404: "Not Found"
default: "default"
type: "string"
translate_when: "/response != null"
- target: "another_translated_result"
regex:
exact: false
patterns:
"2[0-9]{2}" : "Success" # Matches ranges from 200-299
"5[0-9]{2}": "Error" # Matches ranges form 500-599
file:
name: "path/to/file.yaml"
aws:
bucket: my_bucket
region: us-east-1
sts_role_arn: arn:aws:iam::123456789012:role/MyS3Role
```
On the top level, specify `mappings` for inline mapping configurations, or `file` pull mapping configurations from a file. Both `mappings` and `file` options can be specified together, and the processor considers the mappings from both sources for translations. In instances where the pipeline configuration and file mappings share duplicate `source` and `target` pairs, the mappings specified within the pipeline configuration take precedence.
## Configuration
You can use the following options to configure the `translate` processor.
| Parameter | Required | Type | Description |
| :--- | :--- | :--- | :--- |
| mappings | No | List | Defines inline mappings. For more information, see [mappings](#mappings). |
| file | No | Map | Points to the file that contains mapping configurations. For more information, see [file](#file). |
### mappings
Each item in the `mappings` configuration contains the following options.
| Parameter | Required | Type | Description |
| :--- | :--- | :--- | :--- |
| source | Yes | String or list | The source field to translate. Can be a string or a list of strings. |
| targets | Yes | List | A list of target field configurations, such as the target field key or translation maps. |
Each item in the `targets` configuration contains the following options.
| Parameter | Required | Type | Description |
| :--- | :--- | :--- | :--- |
| target | Yes | String | The key that specifies the field in the output in which the translated value will be placed. |
| map | No | Map | A list of key-value pairs that define the translations. Each key represents a possible value in the source field, and the corresponding value represents what it should be translated to. For examples, see [map option](#map-option). At least one of `map` and `regex` should be configured. |
| regex | No | Map | A map of keys that defines the translation map. For more options, see [regex option](#regex-option). At least one of `map` and `regex` should be configured. |
| default | No | String | The default value to use when no match is found during translation. |
| type | No | String | Specifies the data type for the target value. |
| translate_when | No | String | Uses a [Data Prepper expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/) to specify a condition for performing the translation. When specified, the expression will only translate when the condition is met. |
#### map option
You can use the following key types when using the map option:
* Individual keys
```yaml
map:
ok : "Success"
120: "Found"
```
* Number ranges
```yaml
map:
"100-200": "Success"
"400-499": "Error"
```
* Comma-delimited keys
```yaml
map:
"key1,key2,key3": "value1"
"100-200,key4": "value2"
```
When configuring the keys inside the `map` option, do not use any overlapping number ranges or duplicate keys.
#### regex option
You can use the following options with the `regex` option.
| Parameter | Required | Type | Description |
| :--- | :--- | :--- | :--- |
| patterns | Yes | Map | A map of key-value pairs defining the regex patterns of keys and the value to translate to for each pattern. |
| exact | No | Boolean | Whether to use full string match or partial string match on the regex pattern. If `true`, the pattern is considered a match only when the entire key matches the pattern. Otherwise, the pattern is considered a match when a sub-string of the key matches the pattern. |
### file
The `file` option in the `translate` processor takes a local YAML file or an Amazon Simple Storage Service (Amazon S3) object containing translation mappings. The file's contents should be in the following format:
```yaml
mappings:
- source: "status"
targets:
- target: "result"
map:
"foo": "bar"
# Other configurations
```
You can use the following options in the `file` configuration.
| Parameter | Required | Type | Description |
| :--- | :--- | :--- | :--- |
| name | Yes | String | The full path to a local file or key name for an S3 object. |
| aws | No | Map | The AWS configuration when the file is an S3 object. See the following table for more information. |
You can use the following options with the `aws` configuration.
| Parameter | Required | Type | Description |
| :--- | :--- | :--- | :--- |
| `bucket` | Yes | String | The Amazon S3 bucket name. |
| `region` | Yes | String | The AWS Region to use for credentials. |
| `sts_role_arn` | Yes | String | The AWS Security Token Service (AWS STS) role to assume for requests to Amazon S3. |