3.3 KiB

Raw Blame History

layout	title	parent	nav_order
default	Dead-letter queues	Pipelines	13

Dead-letter queues

Data Prepper pipelines support dead-letter queues (DLQs) for offloading failed events and making them accessible for analysis.

As of Data Prepper 2.3, only the s3 source supports DLQs.

Configure a DLQ writer

To configure a DLQ writer for the s3 source, add the following to your pipeline.yaml file:

  sink:
    opensearch:
      dlq:
        s3:
          bucket: "my-dlq-bucket"
          key_path_prefix: "dlq-files/"
          region: "us-west-2"
          sts_role_arn: "arn:aws:iam::123456789012:role/dlq-role"

The resulting DLQ file outputs as a JSON array of DLQ objects. Any file written to the S3 DLQ contains the following name pattern:

dlq-v${version}-${pipelineName}-${pluginId}-${timestampIso8601}-${uniqueId}

The following information is replaced in the name pattern:

version: The Data Prepper version.
pipelineName: The pipeline name indicated in pipeline.yaml.
pluginId: The ID of the plugin associated with the DLQ event.

Configuration

DLQ supports the following configuration options.

Option	Required	Type	Description
bucket	Yes	String	The name of the bucket into which the DLQ outputs failed records.
key_path_prefix	No	String	The `key_prefix` used in the S3 bucket. Defaults to `""`. Supports time value pattern variables, such as `/%{yyyy}/%{MM}/%{dd}`, including any variables listed in the Java DateTimeFormatter. For example, when using the `/%{yyyy}/%{MM}/%{dd}` pattern, you can set `key_prefix` as `/2023/01/24`.
region	No	String	The AWS Region of the S3 bucket. Defaults to `us-east-1`.
sts_role_arn	No	String	The STS role the DLQ assumes in order to write to an AWS S3 bucket. Default is `null`, which uses the standard SDK behavior for credentials. To use this option, the S3 bucket must have the `S3:PutObject` permission configured.

When using DLQ with an OpenSearch sink, you can configure the max_retries option to send failed data to the DLQ when the sink reaches the maximum number of retries.

Metrics

DLQ supports the following metrics.

Counter

dlqS3RecordsSuccess: Measures the number of successful records sent to S3.
dlqS3RecordsFailed: Measures the number of records that failed to be sent to S3.
dlqS3RequestSuccess: Measures the number of successful S3 requests.
dlqS3RequestFailed: Measures the number of failed S3 requests.

Distribution summary

dlqS3RequestSizeBytes: Measures the distribution of the S3 request's payload size in bytes.

Timer

dlqS3RequestLatency: Measures latency when sending each S3 request, including retries.

DLQ objects

DLQ supports the following DLQ objects:

pluginId: The ID of the plugin that originated the event sent to the DLQ.
pluginName: The name of the plugin.
failedData : An object that contains the failed object and its options. This object is unique to each plugin.
pipelineName: The name of the Data Prepper pipeline in which the event failed.
timestamp: The timestamp of the failures in an ISO8601 format.

3.3 KiB Raw Blame History