Naarcha-AWS e1a1f44dd6
Add list to map processor (#3806)
* Add list to map processor.

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Tweak one last file

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Fix typo

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>

* Update mutate-event.md

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

* Add Chris' feedback

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* A couple more wording tweaks

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Update _data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>

---------

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-04-20 13:48:36 -05:00

2.3 KiB

layout title parent grand_parent nav_order
default Trace peer forwarder processors Processors Pipelines 115

trace peer forwarder

The trace peer forwarder processor is used with peer forwarder to reduce by half the number of events forwarded in a Trace Analytics pipeline. In Trace Analytics, each event is typically duplicated when it is sent from otel-trace-pipeline to raw-pipeline and service-map-pipeline. When pipelines forward events, this causes the core peer forwarder to send multiple HTTP requests for the same event. You can use trace peer forwarder to forward an event once through the otel-trace-pipeline instead of raw-pipeline and service-map-pipeline, which prevents unnecessary HTTP requests.

You should use trace peer forwarder for Trace Analytics pipelines when you have multiple nodes.

Usage

To get started with trace peer forwarder, first configure peer forwarder. Then create a pipeline.yaml file and specify trace peer forwarder as the processor. You can configure peer forwarder in your data-prepper-config.yaml file. For more detailed information, see Configuring Data Prepper.

See the following example pipeline.yaml file:

otel-trace-pipeline:
  delay: "100"
  source:
    otel_trace_source:
  processor:
    - trace_peer_forwarder:
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - otel_trace_raw:
  sink:
    - opensearch:
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - service_map_stateful:
  sink:
    - opensearch:

In the preceding pipeline.yaml file, events are forwarded in the otel-trace-pipeline to the target peer, and no forwarding is performed in raw-pipeline or service-map-pipeline. This process helps improve network performance by forwarding events (as HTTP requests) once instead of twice.