mirror of
https://github.com/iSharkFly-Docs/opensearch-docs-cn
synced 2025-03-09 14:38:01 +00:00
e1a1f44dd6
* Add list to map processor. Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Tweak one last file Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Fix typo Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update mutate-event.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Add Chris' feedback Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * A couple more wording tweaks Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> --------- Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
52 lines
3.8 KiB
Markdown
52 lines
3.8 KiB
Markdown
---
|
|
layout: default
|
|
title: Aggregate processor
|
|
parent: Processors
|
|
grand_parent: Pipelines
|
|
nav_order: 41
|
|
---
|
|
|
|
# aggregate
|
|
|
|
The `aggregate` processor groups events based on the keys provided and performs an action on each group.
|
|
|
|
|
|
## Configuration
|
|
|
|
The following table describes the options you can use to configure the `aggregate` processor.
|
|
|
|
Option | Required | Type | Description
|
|
:--- | :--- | :--- | :---
|
|
identification_keys | Yes | List | An unordered list by which to group events. Events with the same values as these keys are put into the same group. If an event does not contain one of the `identification_keys`, then the value of that key is considered to be equal to `null`. At least one identification_key is required (for example, `["sourceIp", "destinationIp", "port"]`).
|
|
action | Yes | AggregateAction | The action to be performed for each group. One of the available aggregate actions must be provided or you can create custom aggregate actions. `remove_duplicates` and `put_all` are the available actions. For more information, see [Creating New Aggregate Actions](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#creating-new-aggregate-actions).
|
|
group_duration | No | String | The amount of time that a group should exist before it is concluded automatically. Supports ISO_8601 notation strings ("PT20.345S", "PT15M", etc.) as well as simple notation for seconds (`"60s"`) and milliseconds (`"1500ms"`). Default value is `180s`.
|
|
|
|
<!---## Configuration
|
|
|
|
Content will be added to this section.--->
|
|
|
|
## Metrics
|
|
|
|
The following table describes common [Abstract processor](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/processor/AbstractProcessor.java) metrics.
|
|
|
|
| Metric name | Type | Description |
|
|
| ------------- | ---- | -----------|
|
|
| `recordsIn` | Counter | Metric representing the ingress of records to a pipeline component. |
|
|
| `recordsOut` | Counter | Metric representing the egress of records from a pipeline component. |
|
|
| `timeElapsed` | Timer | Metric representing the time elapsed during execution of a pipeline component. |
|
|
|
|
|
|
The `aggregate` processor includes the following custom metrics.
|
|
|
|
**Counter**
|
|
|
|
* `actionHandleEventsOut`: The number of events that have been returned from the `handleEvent` call to the configured [action](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#action).
|
|
* `actionHandleEventsDropped`: The number of events that have not been returned from the `handleEvent` call to the configured [action](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#action).
|
|
* `actionHandleEventsProcessingErrors`: The number of calls made to `handleEvent` for the configured [action](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#action) that resulted in an error.
|
|
* `actionConcludeGroupEventsOut`: The number of events that have been returned from the `concludeGroup` call to the configured [action](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#action).
|
|
* `actionConcludeGroupEventsDropped`: The number of events that have not been returned from the `condludeGroup` call to the configured [action](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#action).
|
|
* `actionConcludeGroupEventsProcessingErrors`: The number of calls made to `concludeGroup` for the configured [action](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#action) that resulted in an error.
|
|
|
|
**Gauge**
|
|
|
|
* `currentAggregateGroups`: The current number of groups. This gauge decreases when a group concludes and increases when an event initiates the creation of a new group. |