6.7 KiB

Raw Permalink Blame History

layout	title	parent	grand_parent	nav_order
default	translate	Processors	Pipelines	117

translate

The translate processor transforms values in events into preconfigured values.

Basic usage

To use the translate processor, create the following pipeline.yaml file:

translate-pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - translate:
        mappings:
          - source: "status"
            targets:
              - target: "translated_result"
                map:
                  404: "Not Found"
  sink:
    - stdout:

Then create the following file named logs_json.log and replace the path in the file source of your pipeline.yaml file with the path of a file containing the following JSON data:

{ "status": "404" }

The translate processor configuration in pipeline.yaml retrieves the source value from the event data and compares it against the keys specified under the targets. When a match is found, the processor places the corresponding mapped value into the target key provided in the configuration.

When you run Data Prepper with the previous pipeline.yaml file, you should receive the following output:

{
  "status": "404",
  "translated_result": "Not Found"
}

Advanced options

The following example shows a more involved mapping with additional configurations for the translate processor:

processor:
  - translate:
      mappings:
        - source: "status"
          targets:
            - target: "translated_result"
              map:
                404: "Not Found"
              default: "default"
              type: "string"
              translate_when: "/response != null"
            - target: "another_translated_result"
              regex:
                exact: false
                patterns:
                  "2[0-9]{2}" : "Success" # Matches ranges from 200-299
                  "5[0-9]{2}": "Error"    # Matches ranges form 500-599
      file: 
        name: "path/to/file.yaml"
        aws:
          bucket: my_bucket
          region: us-east-1
          sts_role_arn: arn:aws:iam::123456789012:role/MyS3Role

On the top level, specify mappings for inline mapping configurations, or file pull mapping configurations from a file. Both mappings and file options can be specified together, and the processor considers the mappings from both sources for translations. In instances where the pipeline configuration and file mappings share duplicate source and target pairs, the mappings specified within the pipeline configuration take precedence.

Configuration

You can use the following options to configure the translate processor.

Parameter	Required	Type	Description
mappings	No	List	Defines inline mappings. For more information, see mappings.
file	No	Map	Points to the file that contains mapping configurations. For more information, see file.

mappings

Each item in the mappings configuration contains the following options.

Parameter	Required	Type	Description
source	Yes	String or list	The source field to translate. Can be a string or a list of strings.
targets	Yes	List	A list of target field configurations, such as the target field key or translation maps.

Each item in the targets configuration contains the following options.

Parameter	Required	Type	Description
target	Yes	String	The key that specifies the field in the output in which the translated value will be placed.
map	No	Map	A list of key-value pairs that define the translations. Each key represents a possible value in the source field, and the corresponding value represents what it should be translated to. For examples, see map option. At least one of `map` and `regex` should be configured.
regex	No	Map	A map of keys that defines the translation map. For more options, see regex option. At least one of `map` and `regex` should be configured.
default	No	String	The default value to use when no match is found during translation.
type	No	String	Specifies the data type for the target value.
translate_when	No	String	Uses a Data Prepper expression to specify a condition for performing the translation. When specified, the expression will only translate when the condition is met.

map option

You can use the following key types when using the map option:

Individual keys

  map:
    ok : "Success"
    120: "Found"

Number ranges

  map:
    "100-200": "Success"
    "400-499": "Error"

Comma-delimited keys

  map:
    "key1,key2,key3": "value1"
    "100-200,key4": "value2"

When configuring the keys inside the map option, do not use any overlapping number ranges or duplicate keys.

regex option

You can use the following options with the regex option.

Parameter	Required	Type	Description
patterns	Yes	Map	A map of key-value pairs defining the regex patterns of keys and the value to translate to for each pattern.
exact	No	Boolean	Whether to use full string match or partial string match on the regex pattern. If `true`, the pattern is considered a match only when the entire key matches the pattern. Otherwise, the pattern is considered a match when a sub-string of the key matches the pattern.

file

The file option in the translate processor takes a local YAML file or an Amazon Simple Storage Service (Amazon S3) object containing translation mappings. The file's contents should be in the following format:

mappings:
  - source: "status"
    targets:
      - target: "result"
        map:
          "foo": "bar"
        # Other configurations

You can use the following options in the file configuration.

Parameter	Required	Type	Description
name	Yes	String	The full path to a local file or key name for an S3 object.
aws	No	Map	The AWS configuration when the file is an S3 object. See the following table for more information.

You can use the following options with the aws configuration.

Parameter	Required	Type	Description
`bucket`	Yes	String	The Amazon S3 bucket name.
`region`	Yes	String	The AWS Region to use for credentials.
`sts_role_arn`	Yes	String	The AWS Security Token Service (AWS STS) role to assume for requests to Amazon S3.

6.7 KiB Raw Permalink Blame History