Caroline 8eac00ec8f
Add Trace peer forwarder doc. (#3238)
* Add trace peer forwarder doc.

Signed-off-by: carolxob <carolxob@amazon.com>

* Updated file title.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor updates based on tech review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Fixed header info and links.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor updates.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor updates.

Signed-off-by: carolxob <carolxob@amazon.com>

* Added questions as comments to ask SME.

Signed-off-by: carolxob <carolxob@amazon.com>

* Added Trace analytics link.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor update.

Signed-off-by: carolxob <carolxob@amazon.com>

* Removed markdown comments and added them in GitHub.

Signed-off-by: carolxob <carolxob@amazon.com>

* Made updates based on tech review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Removed markdown comment.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edits.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor change to unnecessary punctuation.

Signed-off-by: carolxob <carolxob@amazon.com>

* Some feedback from doc review.

Signed-off-by: carolxob <carolxob@amazon.com>

* Fixed broken links.

Signed-off-by: carolxob <carolxob@amazon.com>

* Fixed links.

Signed-off-by: carolxob <carolxob@amazon.com>

* Updates based on editorial feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Update _data-prepper/pipelines/configuration/processors/trace-peer-forwarder.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Update _data-prepper/pipelines/configuration/processors/trace-peer-forwarder.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Fixed hyperlinks to include data-prepper in path for peer forwarder.

Signed-off-by: carolxob <carolxob@amazon.com>

---------

Signed-off-by: carolxob <carolxob@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-04-11 11:35:19 -06:00

2.3 KiB

layout title parent grand_parent nav_order
default trace peer forwarder Processors Pipelines 45

trace peer forwarder

The trace peer forwarder processor is used with peer forwarder to reduce by half the number of events forwarded in a Trace Analytics pipeline. In Trace Analytics, each event is typically duplicated when it is sent from otel-trace-pipeline to raw-pipeline and service-map-pipeline. When pipelines forward events, this causes the core peer forwarder to send multiple HTTP requests for the same event. You can use trace peer forwarder to forward an event once through the otel-trace-pipeline instead of raw-pipeline and service-map-pipeline, which prevents unnecessary HTTP requests.

You should use trace peer forwarder for Trace Analytics pipelines when you have multiple nodes.

Usage

To get started with trace peer forwarder, first configure peer forwarder. Then create a pipeline.yaml file and specify trace peer forwarder as the processor. You can configure peer forwarder in your data-prepper-config.yaml file. For more detailed information, see Configuring Data Prepper.

See the following example pipeline.yaml file:

otel-trace-pipeline:
  delay: "100"
  source:
    otel_trace_source:
  processor:
    - trace_peer_forwarder:
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - otel_trace_raw:
  sink:
    - opensearch:
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - service_map_stateful:
  sink:
    - opensearch:

In the preceding pipeline.yaml file, events are forwarded in the otel-trace-pipeline to the target peer, and no forwarding is performed in raw-pipeline or service-map-pipeline. This process helps improve network performance by forwarding events (as HTTP requests) once instead of twice.