* Add trace peer forwarder doc. Signed-off-by: carolxob <carolxob@amazon.com> * Updated file title. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates based on tech review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Fixed header info and links. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates. Signed-off-by: carolxob <carolxob@amazon.com> * Added questions as comments to ask SME. Signed-off-by: carolxob <carolxob@amazon.com> * Added Trace analytics link. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update. Signed-off-by: carolxob <carolxob@amazon.com> * Removed markdown comments and added them in GitHub. Signed-off-by: carolxob <carolxob@amazon.com> * Made updates based on tech review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Removed markdown comment. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Minor change to unnecessary punctuation. Signed-off-by: carolxob <carolxob@amazon.com> * Some feedback from doc review. Signed-off-by: carolxob <carolxob@amazon.com> * Fixed broken links. Signed-off-by: carolxob <carolxob@amazon.com> * Fixed links. Signed-off-by: carolxob <carolxob@amazon.com> * Updates based on editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Update _data-prepper/pipelines/configuration/processors/trace-peer-forwarder.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Update _data-prepper/pipelines/configuration/processors/trace-peer-forwarder.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Fixed hyperlinks to include data-prepper in path for peer forwarder. Signed-off-by: carolxob <carolxob@amazon.com> --------- Signed-off-by: carolxob <carolxob@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2.3 KiB
layout | title | parent | grand_parent | nav_order |
---|---|---|---|---|
default | trace peer forwarder | Processors | Pipelines | 45 |
trace peer forwarder
The trace peer forwarder
processor is used with peer forwarder to reduce by half the number of events forwarded in a Trace Analytics pipeline. In Trace Analytics, each event is typically duplicated when it is sent from otel-trace-pipeline
to raw-pipeline
and service-map-pipeline
. When pipelines forward events, this causes the core peer forwarder to send multiple HTTP requests for the same event. You can use trace peer forwarder
to forward an event once through the otel-trace-pipeline
instead of raw-pipeline
and service-map-pipeline
, which prevents unnecessary HTTP requests.
You should use trace peer forwarder
for Trace Analytics pipelines when you have multiple nodes.
Usage
To get started with trace peer forwarder
, first configure peer forwarder. Then create a pipeline.yaml
file and specify trace peer forwarder
as the processor. You can configure peer forwarder
in your data-prepper-config.yaml
file. For more detailed information, see Configuring Data Prepper.
See the following example pipeline.yaml
file:
otel-trace-pipeline:
delay: "100"
source:
otel_trace_source:
processor:
- trace_peer_forwarder:
sink:
- pipeline:
name: "raw-pipeline"
- pipeline:
name: "service-map-pipeline"
raw-pipeline:
source:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
sink:
- opensearch:
service-map-pipeline:
delay: "100"
source:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
sink:
- opensearch:
In the preceding pipeline.yaml
file, events are forwarded in the otel-trace-pipeline
to the target peer, and no forwarding is performed in raw-pipeline
or service-map-pipeline
. This process helps improve network performance by forwarding events (as HTTP requests) once instead of twice.