Naarcha-AWS e1a1f44dd6
Add list to map processor (#3806)
* Add list to map processor.

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Tweak one last file

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Fix typo

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>

* Update mutate-event.md

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

* Add Chris' feedback

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* A couple more wording tweaks

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>

---------

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-04-20 13:48:36 -05:00

2.0 KiB

layout, title, parent, grand_parent, nav_order
layout title parent grand_parent nav_order
default Add entries processor Processors Pipelines 40

add_entries

The add_entries processor adds entries to an event.

Configuration

You can configure the add_entries processor with the following options.

Option Required Description
entries Yes A list of entries to add to an event.
key Yes The key of the new entry to be added. Some examples of keys include my_key, myKey, and object/sub_Key.
value Yes The value of the new entry to be added. You can use the following data types: strings, Booleans, numbers, null, nested objects, and arrays.
overwrite_if_key_exists No When set to true, the existing value is overwritten if key already exists in the event. The default value is false.

Usage

To get started, create the following pipeline.yaml file:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - add_entries:
        entries:
        - key: "newMessage"
          value: 3
          overwrite_if_key_exists: true
  sink:
    - stdout:

{% include copy.html %}

Next, create a log file named logs_json.log and replace the path in the file source of your pipeline.yaml file with that filepath. For more information, see Configuring Data Prepper.

For example, before you run the add_entries processor, if the logs_json.log file contains the following event record:

{"message": "hello"}

Then when you run the add_entries processor using the previous configuration, it adds a new entry {"newMessage": 3} to the existing event {"message": "hello"} so that the new event contains two entries in the final output:

{"message": "hello", "newMessage": 3}

If newMessage already exists, its existing value is overwritten with a value of 3.