opensearch-docs-cn/_data-prepper/pipelines/configuration/processors/date.md

44 lines
3.5 KiB
Markdown
Raw Normal View History

Restructure Data Prepper plugins documentation (#2073) * Removed content from Data Prepper reference and broke out into separate pages. Signed-off-by: carolxob <carolxob@amazon.com> * Checking in file to make sure it's the right version. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update. Signed-off-by: carolxob <carolxob@amazon.com> * Updated files. Signed-off-by: carolxob <carolxob@amazon.com> * Adding Sinks file. Signed-off-by: carolxob <carolxob@amazon.com> * Added file to PR. Signed-off-by: carolxob <carolxob@amazon.com> * Corrected TOC hierarchy. Signed-off-by: carolxob <carolxob@amazon.com> * Added images and reorganized files. Signed-off-by: carolxob <carolxob@amazon.com> * Reconfigured some content based on David's feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Modified reference page. Signed-off-by: carolxob <carolxob@amazon.com> * Fixed minor heading issue. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Major edits, files created, moved, and content broken out from main config page. Signed-off-by: carolxob <carolxob@amazon.com> * Adding key value and processors pages to the PR. Signed-off-by: carolxob <carolxob@amazon.com> * Basic ToC reorg. Signed-off-by: carolxob <carolxob@amazon.com> * ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update.: Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits to ToC again. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor TOC update Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC edits. Signed-off-by: carolxob <carolxob@amazon.com> * Changed filename. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates Signed-off-by: carolxob <carolxob@amazon.com> * Making small Toc changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Added comment blocks for Config and Metrics sections. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates to add Sinks and Sources under config guide. Signed-off-by: carolxob <carolxob@amazon.com> Signed-off-by: carolxob <carolxob@amazon.com>
2022-12-27 17:58:48 -05:00
---
layout: default
Add DLQ and aggregate processor edits. Baseline processor names (#3880) * Add DLQ and aggregate processor edits. Baseline processor names Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Remove processor from titles Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Update dlq.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Add additional feedback Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * add cross links Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Fix typos Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Add doc review Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Remove link Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Apply suggestions from code review Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update dlq.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Fix processor title Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-04-27 15:01:01 -04:00
title: date
Restructure Data Prepper plugins documentation (#2073) * Removed content from Data Prepper reference and broke out into separate pages. Signed-off-by: carolxob <carolxob@amazon.com> * Checking in file to make sure it's the right version. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update. Signed-off-by: carolxob <carolxob@amazon.com> * Updated files. Signed-off-by: carolxob <carolxob@amazon.com> * Adding Sinks file. Signed-off-by: carolxob <carolxob@amazon.com> * Added file to PR. Signed-off-by: carolxob <carolxob@amazon.com> * Corrected TOC hierarchy. Signed-off-by: carolxob <carolxob@amazon.com> * Added images and reorganized files. Signed-off-by: carolxob <carolxob@amazon.com> * Reconfigured some content based on David's feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Modified reference page. Signed-off-by: carolxob <carolxob@amazon.com> * Fixed minor heading issue. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Major edits, files created, moved, and content broken out from main config page. Signed-off-by: carolxob <carolxob@amazon.com> * Adding key value and processors pages to the PR. Signed-off-by: carolxob <carolxob@amazon.com> * Basic ToC reorg. Signed-off-by: carolxob <carolxob@amazon.com> * ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update.: Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits to ToC again. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor TOC update Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC edits. Signed-off-by: carolxob <carolxob@amazon.com> * Changed filename. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates Signed-off-by: carolxob <carolxob@amazon.com> * Making small Toc changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Added comment blocks for Config and Metrics sections. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates to add Sinks and Sources under config guide. Signed-off-by: carolxob <carolxob@amazon.com> Signed-off-by: carolxob <carolxob@amazon.com>
2022-12-27 17:58:48 -05:00
parent: Processors
Data Prepper ToC Update (#2514) * Creating PR with first file. Signed-off-by: carolxob <carolxob@amazon.com> * Adding newly created files to PR. Signed-off-by: carolxob <carolxob@amazon.com> * Reorganized files and added appropriate metadata to map ToC correctly. Signed-off-by: carolxob <carolxob@amazon.com> * Moved Authoring pipelines page. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates to Sources section for Data Prepper. Signed-off-by: carolxob <carolxob@amazon.com> * Updated Buffers section under Data Prepper. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update to otelmetricssource. Signed-off-by: carolxob <carolxob@amazon.com> * Restructured ToC in Processors section for Data Prepper. Signed-off-by: carolxob <carolxob@amazon.com> * Minor filename change. Signed-off-by: carolxob <carolxob@amazon.com> * Adjustments to metadata in ToC. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edit. Signed-off-by: carolxob <carolxob@amazon.com> * Fixed nav order in metadata. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edit. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update top metadata for ToC. Signed-off-by: carolxob <carolxob@amazon.com> * Adjustmenets to Toc order. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustments to ToC metadata. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustments to Sinks section. Signed-off-by: carolxob <carolxob@amazon.com> * Adjustements to high level ToC. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustement to Pipelines.md Signed-off-by: carolxob <carolxob@amazon.com> * Minor update. Signed-off-by: carolxob <carolxob@amazon.com> * Slight reorganization. Removed two placeholder pages for now. Signed-off-by: carolxob <carolxob@amazon.com> * Removed a page and replaced with pipelines content. Signed-off-by: carolxob <carolxob@amazon.com> * Minor changes/additions to content for placeholder pages. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update to page link. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustments to ToC metadata. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Removed /clients from redirects to correct nav order. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustments to ToC metadata. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustments. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustment ot metadata. Signed-off-by: carolxob <carolxob@amazon.com> * TOC link fixes Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Changed page name. Signed-off-by: carolxob <carolxob@amazon.com> * Corrected references to Peer Forwarder. Signed-off-by: carolxob <carolxob@amazon.com> * Renamed Data Prepper folder. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates to phrasing and capitalization. Signed-off-by: carolxob <carolxob@amazon.com> * Minor phrasing update. Signed-off-by: carolxob <carolxob@amazon.com> * Minor phrasing update. Signed-off-by: carolxob <carolxob@amazon.com> * Minor change. Signed-off-by: carolxob <carolxob@amazon.com> * Minor change to change S3 Source to S3Source. Signed-off-by: carolxob <carolxob@amazon.com> * Updated references to peer forwarder and changed capitalization. Signed-off-by: carolxob <carolxob@amazon.com> * Updated capitalization for peer forwarder. Signed-off-by: carolxob <carolxob@amazon.com> * Made edits based on doc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Update to one word. Signed-off-by: carolxob <carolxob@amazon.com> --------- Signed-off-by: carolxob <carolxob@amazon.com> Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Co-authored-by: Naarcha-AWS <naarcha@amazon.com>
2023-02-03 17:06:10 -05:00
grand_parent: Pipelines
Add list to map processor (#3806) * Add list to map processor. Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Tweak one last file Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Fix typo Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update mutate-event.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Add Chris' feedback Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * A couple more wording tweaks Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> --------- Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-04-20 14:48:36 -04:00
nav_order: 50
Restructure Data Prepper plugins documentation (#2073) * Removed content from Data Prepper reference and broke out into separate pages. Signed-off-by: carolxob <carolxob@amazon.com> * Checking in file to make sure it's the right version. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update. Signed-off-by: carolxob <carolxob@amazon.com> * Updated files. Signed-off-by: carolxob <carolxob@amazon.com> * Adding Sinks file. Signed-off-by: carolxob <carolxob@amazon.com> * Added file to PR. Signed-off-by: carolxob <carolxob@amazon.com> * Corrected TOC hierarchy. Signed-off-by: carolxob <carolxob@amazon.com> * Added images and reorganized files. Signed-off-by: carolxob <carolxob@amazon.com> * Reconfigured some content based on David's feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Modified reference page. Signed-off-by: carolxob <carolxob@amazon.com> * Fixed minor heading issue. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Major edits, files created, moved, and content broken out from main config page. Signed-off-by: carolxob <carolxob@amazon.com> * Adding key value and processors pages to the PR. Signed-off-by: carolxob <carolxob@amazon.com> * Basic ToC reorg. Signed-off-by: carolxob <carolxob@amazon.com> * ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update.: Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits to ToC again. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor TOC update Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC edits. Signed-off-by: carolxob <carolxob@amazon.com> * Changed filename. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates Signed-off-by: carolxob <carolxob@amazon.com> * Making small Toc changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Added comment blocks for Config and Metrics sections. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates to add Sinks and Sources under config guide. Signed-off-by: carolxob <carolxob@amazon.com> Signed-off-by: carolxob <carolxob@amazon.com>
2022-12-27 17:58:48 -05:00
---
# date
Add list to map processor (#3806) * Add list to map processor. Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Tweak one last file Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Fix typo Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> * Update mutate-event.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Add Chris' feedback Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * A couple more wording tweaks Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> --------- Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-04-20 14:48:36 -04:00
The `date` processor adds a default timestamp to an event, parses timestamp fields, and converts timestamp information to the International Organization for Standardization (ISO) 8601 format. This timestamp information can be used as an event timestamp.
## Configuration
The following table describes the options you can use to configure the `date` processor.
Restructure Data Prepper plugins documentation (#2073) * Removed content from Data Prepper reference and broke out into separate pages. Signed-off-by: carolxob <carolxob@amazon.com> * Checking in file to make sure it's the right version. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update. Signed-off-by: carolxob <carolxob@amazon.com> * Updated files. Signed-off-by: carolxob <carolxob@amazon.com> * Adding Sinks file. Signed-off-by: carolxob <carolxob@amazon.com> * Added file to PR. Signed-off-by: carolxob <carolxob@amazon.com> * Corrected TOC hierarchy. Signed-off-by: carolxob <carolxob@amazon.com> * Added images and reorganized files. Signed-off-by: carolxob <carolxob@amazon.com> * Reconfigured some content based on David's feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Modified reference page. Signed-off-by: carolxob <carolxob@amazon.com> * Fixed minor heading issue. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Major edits, files created, moved, and content broken out from main config page. Signed-off-by: carolxob <carolxob@amazon.com> * Adding key value and processors pages to the PR. Signed-off-by: carolxob <carolxob@amazon.com> * Basic ToC reorg. Signed-off-by: carolxob <carolxob@amazon.com> * ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update.: Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits to ToC again. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor TOC update Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC edits. Signed-off-by: carolxob <carolxob@amazon.com> * Changed filename. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates Signed-off-by: carolxob <carolxob@amazon.com> * Making small Toc changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Added comment blocks for Config and Metrics sections. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates to add Sinks and Sources under config guide. Signed-off-by: carolxob <carolxob@amazon.com> Signed-off-by: carolxob <carolxob@amazon.com>
2022-12-27 17:58:48 -05:00
Option | Required | Type | Description
:--- | :--- | :--- | :---
match | Conditionally | List | List of `key` and `patterns` where patterns is a list. The list of match can have exactly one `key` and `patterns`. There is no default value. This option cannot be defined at the same time as `from_time_received`. Include multiple date processors in your pipeline if both options should be used.
from_time_received | Conditionally | Boolean | A boolean that is used for adding default timestamp to event data from event metadata which is the time when source receives the event. Default value is `false`. This option cannot be defined at the same time as `match`. Include multiple date processors in your pipeline if both options should be used.
destination | No | String | Field to store the timestamp parsed by date processor. It can be used with both `match` and `from_time_received`. Default value is `@timestamp`.
source_timezone | No | String | Time zone used to parse dates. It is used in case the zone or offset cannot be extracted from the value. If the zone or offset are part of the value, then timezone is ignored. Find all the available timezones [the list of database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) in the **TZ database name** column.
destination_timezone | No | String | Timezone used for storing timestamp in `destination` field. The available timezone values are the same as `source_timestamp`.
locale | No | String | Locale is used for parsing dates. It's commonly used for parsing month names(`MMM`). It can have language, country and variant fields using IETF BCP 47 or String representation of [Locale](https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html) object. For example `en-US` for IETF BCP 47 and `en_US` for string representation of Locale. Full list of locale fields which includes language, country and variant can be found [the language subtag registry](https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry). Default value is `Locale.ROOT`.
<!---## Configuration
Added metrics sections to Aggregate processor subpages. (#2730) * Added metrics section to Aggregate processor page. Signed-off-by: carolxob <carolxob@amazon.com> * Added Metrics section to individual Processors pages. Signed-off-by: carolxob <carolxob@amazon.com> * Added metrics section for JSON processor. Signed-off-by: carolxob <carolxob@amazon.com> * Added metrics sections. Changed Default is to Default value is. Signed-off-by: carolxob <carolxob@amazon.com> * Corrected references from AWS S3 to Amazon S3. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates to Metrics sections and phrasing. Signed-off-by: carolxob <carolxob@amazon.com> * Updated Action link. Signed-off-by: carolxob <carolxob@amazon.com> * Updates based on tech review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Updates based on tech review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Tech review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates to buffer_size and batch_size default values. Signed-off-by: carolxob <carolxob@amazon.com> * Edits to Metrics sections for each processor. Signed-off-by: carolxob <carolxob@amazon.com> * Update made based ondoc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates to intro text for processor pages. Minor adjustements to other text for clarity. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Adjustements to phrasing, fixed typos. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates to word choice and corrected a typo. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edit. Signed-off-by: carolxob <carolxob@amazon.com> * Made updates based ondoc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Updates to http-source. Signed-off-by: carolxob <carolxob@amazon.com> * Added common processors table to affected docs. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update to one file. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update based on tech review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Major editorial feedback incorporated through key-value.md. Signed-off-by: carolxob <carolxob@amazon.com> * Incorporated major editorial feedback thup to service-map-stateful. Signed-off-by: carolxob <carolxob@amazon.com> * Incorporated major editorial feedback for Processors section. Signed-off-by: carolxob <carolxob@amazon.com> * Major editorial updates, specifically to inclusion of text introducing option configuration tables. Signed-off-by: carolxob <carolxob@amazon.com> * Major editorial feedback through otel-trace.md incorporated. Signed-off-by: carolxob <carolxob@amazon.com> * Major editorial edits incorporated. Signed-off-by: carolxob <carolxob@amazon.com> * Technical feedback and editorial feedback incorporated. Signed-off-by: carolxob <carolxob@amazon.com> * Incorporated missing editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustements to OpenSearch sink. Signed-off-by: carolxob <carolxob@amazon.com> * Minor changes to capitalization. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Made one instance of processor name consistent with other references. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update based on editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> --------- Signed-off-by: carolxob <carolxob@amazon.com>
2023-02-27 11:40:59 -05:00
Content will be added to this section.--->
Restructure Data Prepper plugins documentation (#2073) * Removed content from Data Prepper reference and broke out into separate pages. Signed-off-by: carolxob <carolxob@amazon.com> * Checking in file to make sure it's the right version. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update. Signed-off-by: carolxob <carolxob@amazon.com> * Updated files. Signed-off-by: carolxob <carolxob@amazon.com> * Adding Sinks file. Signed-off-by: carolxob <carolxob@amazon.com> * Added file to PR. Signed-off-by: carolxob <carolxob@amazon.com> * Corrected TOC hierarchy. Signed-off-by: carolxob <carolxob@amazon.com> * Added images and reorganized files. Signed-off-by: carolxob <carolxob@amazon.com> * Reconfigured some content based on David's feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Modified reference page. Signed-off-by: carolxob <carolxob@amazon.com> * Fixed minor heading issue. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Major edits, files created, moved, and content broken out from main config page. Signed-off-by: carolxob <carolxob@amazon.com> * Adding key value and processors pages to the PR. Signed-off-by: carolxob <carolxob@amazon.com> * Basic ToC reorg. Signed-off-by: carolxob <carolxob@amazon.com> * ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update.: Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits to ToC again. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor TOC update Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC edits. Signed-off-by: carolxob <carolxob@amazon.com> * Changed filename. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates Signed-off-by: carolxob <carolxob@amazon.com> * Making small Toc changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Added comment blocks for Config and Metrics sections. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates to add Sinks and Sources under config guide. Signed-off-by: carolxob <carolxob@amazon.com> Signed-off-by: carolxob <carolxob@amazon.com>
2022-12-27 17:58:48 -05:00
## Metrics
Added metrics sections to Aggregate processor subpages. (#2730) * Added metrics section to Aggregate processor page. Signed-off-by: carolxob <carolxob@amazon.com> * Added Metrics section to individual Processors pages. Signed-off-by: carolxob <carolxob@amazon.com> * Added metrics section for JSON processor. Signed-off-by: carolxob <carolxob@amazon.com> * Added metrics sections. Changed Default is to Default value is. Signed-off-by: carolxob <carolxob@amazon.com> * Corrected references from AWS S3 to Amazon S3. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates to Metrics sections and phrasing. Signed-off-by: carolxob <carolxob@amazon.com> * Updated Action link. Signed-off-by: carolxob <carolxob@amazon.com> * Updates based on tech review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Updates based on tech review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Tech review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates to buffer_size and batch_size default values. Signed-off-by: carolxob <carolxob@amazon.com> * Edits to Metrics sections for each processor. Signed-off-by: carolxob <carolxob@amazon.com> * Update made based ondoc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates to intro text for processor pages. Minor adjustements to other text for clarity. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Adjustements to phrasing, fixed typos. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates to word choice and corrected a typo. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edit. Signed-off-by: carolxob <carolxob@amazon.com> * Made updates based ondoc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Updates to http-source. Signed-off-by: carolxob <carolxob@amazon.com> * Added common processors table to affected docs. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update to one file. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update based on tech review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Major editorial feedback incorporated through key-value.md. Signed-off-by: carolxob <carolxob@amazon.com> * Incorporated major editorial feedback thup to service-map-stateful. Signed-off-by: carolxob <carolxob@amazon.com> * Incorporated major editorial feedback for Processors section. Signed-off-by: carolxob <carolxob@amazon.com> * Major editorial updates, specifically to inclusion of text introducing option configuration tables. Signed-off-by: carolxob <carolxob@amazon.com> * Major editorial feedback through otel-trace.md incorporated. Signed-off-by: carolxob <carolxob@amazon.com> * Major editorial edits incorporated. Signed-off-by: carolxob <carolxob@amazon.com> * Technical feedback and editorial feedback incorporated. Signed-off-by: carolxob <carolxob@amazon.com> * Incorporated missing editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustements to OpenSearch sink. Signed-off-by: carolxob <carolxob@amazon.com> * Minor changes to capitalization. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Made one instance of processor name consistent with other references. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update based on editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> --------- Signed-off-by: carolxob <carolxob@amazon.com>
2023-02-27 11:40:59 -05:00
The following table describes common [Abstract processor](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/processor/AbstractProcessor.java) metrics.
| Metric name | Type | Description |
| ------------- | ---- | -----------|
| `recordsIn` | Counter | Metric representing the ingress of records to a pipeline component. |
| `recordsOut` | Counter | Metric representing the egress of records from a pipeline component. |
| `timeElapsed` | Timer | Metric representing the time elapsed during execution of a pipeline component. |
The `date` processor includes the following custom metrics.
* `dateProcessingMatchSuccessCounter`: Returns the number of records that match with at least one pattern specified by the `match configuration` option.
* `dateProcessingMatchFailureCounter`: Returns the number of records that did not match any of the patterns specified by the `patterns match` configuration option.