5b5b1d0de1
Add metrics traces use case to Data Prepper ( #6251 )
...
* Add metrics traces use case to Data Prepper
---------
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
2024-02-22 14:49:58 -07:00
83fd43c4d4
Update CSV Data Prepper processor ( #6378 )
...
* Update csv.md
Move example usage from plugin README: 4c4677bc6a/data-prepper-plugins/csv-processor/README.md
2024-02-22 11:42:32 -06:00
d20468bb30
Add event aggregation use case to Data Prepper documentation ( #6206 )
...
* Add event aggregation use case to Data Prepper documentation
---------
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2024-02-22 10:20:37 -07:00
a079c51e88
Add sampling with Data Prepper use case ( #6186 )
...
* Add sampling with Data Prepper use case
---------
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
2024-02-22 10:16:50 -07:00
74e50ca9fc
MAINT: Data Prepper opensearch source metrics ( #6392 )
...
* MAINT: opensearch source metrics
Signed-off-by: George Chen <qchea@amazon.com>
* MNT: rewording
Signed-off-by: George Chen <qchea@amazon.com>
* MAINT: correct typo and rewording
Signed-off-by: George Chen <qchea@amazon.com>
* Update _data-prepper/pipelines/configuration/sources/opensearch.md
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Qi Chen <chenqi0805@gmail.com>
* Update _data-prepper/pipelines/configuration/sources/opensearch.md
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Qi Chen <chenqi0805@gmail.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Qi Chen <chenqi0805@gmail.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2024-02-16 12:08:49 -06:00
fedb5504df
Add use case for anomaly detection with Data Prepper ( #6140 )
...
* Add anomaly detection use case for Data Prepper
---------
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2024-02-14 14:44:31 -07:00
d9d5406f93
Add pattern matching (text processing) use case to Data Prepper documentation ( #6181 )
...
* Add pattern matching use case to Data Prepper documentation
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Add pattern matching use case to Data Prepper documentation
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Address SME comments
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Copy edits
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Address editorial feedback
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
---------
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
2024-01-30 12:00:58 -07:00
719b3895cf
Add log enrichment use case to Data Prepper ( #6225 )
...
* Add log enrichment use case to Data Prepper
---------
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2024-01-22 13:15:47 -07:00
699cb9ac20
Change navigation for index pages of collections ( #6039 )
...
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
2024-01-04 14:55:05 -05:00
a7da03343e
Style guide fix - white space ( #6023 )
...
* Style guide fix - white space
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
* Update _data-prepper/pipelines/configuration/processors/parse-json.md
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
---------
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2024-01-02 14:27:35 -05:00
e63b67b873
Added documentation for grok_when option ( #5912 )
...
* Added documentation for grok_when option
Signed-off-by: Kondaka <krishkdk@amazon.com>
* Apply suggestions from code review
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/processors/grok.md
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Update _data-prepper/pipelines/configuration/processors/grok.md
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Update _data-prepper/pipelines/configuration/processors/grok.md
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Update _data-prepper/pipelines/configuration/processors/grok.md
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Update _data-prepper/pipelines/configuration/processors/grok.md
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Update _data-prepper/pipelines/configuration/processors/grok.md
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Update grok.md
Address editorial review feedback.
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
* Update grok.md
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
---------
Signed-off-by: Kondaka <krishkdk@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-12-21 13:39:11 -07:00
21de57685c
Documentation for Data Prepper's kafka buffer ( #5689 )
...
* Adds documentation for Data Prepper's kafka buffer.
Signed-off-by: David Venable <dlv@amazon.com>
* Update kafka.md
* Adds an example pipeline configuration for the Kafka buffer in Data Prepper. Also some related clean-up.
Signed-off-by: David Venable <dlv@amazon.com>
* Update _data-prepper/pipelines/configuration/buffers/kafka.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/buffers/kafka.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-12-13 14:12:26 -06:00
9c4e13bd2e
Fix missing character typo ( #5771 )
...
* Fix missing character typo
Signed-off-by: Craig Adam <craig@adam11.ca>
* Update substitute-string.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Craig Adam <craig@adam11.ca>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-12-04 12:27:39 -06:00
fa5bc07a69
Add DynamoDB source ( #5664 )
...
* Add DynamoDB source
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Reconfigure sections
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Add Dynamo DB source.
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update dynamo-db.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update dynamo-db.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update dynamo-db.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update dynamo-db.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-11-29 11:38:08 -06:00
5877f13f16
Add Serverless configuration options for Data Prepper ( #5663 )
...
* Add Serverless configuration options for Data Prepper
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Implement additional technical feedback
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Fix options
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/sinks/opensearch.md
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-11-28 10:10:15 -06:00
97d14ccfc2
Data Prepper's s3 source: visibility_duplication_protection ( #5665 )
...
* Adds documentation for Data Prepper's s3 source visibility_duplication_protection configuration. Includes the necessary permissions.
Signed-off-by: David Venable <dlv@amazon.com>
* Update _data-prepper/pipelines/configuration/sources/s3.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/sources/s3.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-11-27 11:26:54 -06:00
72b3363460
Add codec and processor combinations to S3 source ( #5288 )
...
* Add codec and processor combinations to S3 source.
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Update s3.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Make codec-processor combinations its own file
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Make Data Prepper codec page
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Fix links
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Fix links
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Reformat article without tables.
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* fix link
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/common-use-cases/codec-processor-combinations.md
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update codec-processor-combinations.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
2023-11-17 20:06:04 -06:00
e4c5ec18db
[DOC] Fix rename processor 'otel_trace' to 'otel_traces' ( #5423 )
...
* Fix APIs File
Signed-off-by: samipak458 <samipak458@gmail.com>
* Update API files according to API template
Signed-off-by: samipak458 <samipak458@gmail.com>
* Update exec-stored-script.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update cat-aliases.md
Signed-off-by: MUHAMMAD SAMIULLAH <samipak458@gmail.com>
* Update cat-allocation.md
Signed-off-by: MUHAMMAD SAMIULLAH <samipak458@gmail.com>
* Update cat-cluster_manager.md
Signed-off-by: MUHAMMAD SAMIULLAH <samipak458@gmail.com>
* Update cat-count.md
Signed-off-by: MUHAMMAD SAMIULLAH <samipak458@gmail.com>
* Update cat-field-data.md
Signed-off-by: MUHAMMAD SAMIULLAH <samipak458@gmail.com>
* Update cat-health.md
Signed-off-by: MUHAMMAD SAMIULLAH <samipak458@gmail.com>
* updated
* updated
* updated
* data prepper update
Signed-off-by: samipak458 <samipak458@gmail.com>
* Update _data-prepper/managing-data-prepper/peer-forwarder.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: samipak458 <samipak458@gmail.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: MUHAMMAD SAMIULLAH <samipak458@gmail.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-11-07 14:48:11 -06:00
3c94bc80d7
Fix links and liquid formatting error ( #5512 )
...
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
2023-11-06 12:23:59 -05:00
96175482d6
[DOC] data prepper secret extensions ( #5202 )
...
* ADD: extension docs in data-prepper-config
Signed-off-by: George Chen <qchea@amazon.com>
* MAINT: updating secrets extension doc
Signed-off-by: George Chen <qchea@amazon.com>
* MAINT: fix links
Signed-off-by: George Chen <qchea@amazon.com>
* MAINT: fix one more dead link
Signed-off-by: George Chen <qchea@amazon.com>
* MAINT: renaming
Signed-off-by: George Chen <qchea@amazon.com>
* Update configuring-data-prepper.md
* Update pipelines.md
---------
Signed-off-by: George Chen <qchea@amazon.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-11-03 12:08:00 -05:00
7012af124a
[OSCI][DOCs] Replacing 'indices' terms for 'indexes' terms ONLY for description texts ( #5353 )
...
* Fixing documentation for Wildcard in term-level queries section for Query DSL
Signed-off-by: Samuel Valdes Gutierrez <valdesgutierrez@gmail.com>
* replacing 'indices' term for 'indexes' term ONLY for description texts (not variables, links or properties)
Signed-off-by: Samuel Valdes Gutierrez <valdesgutierrez@gmail.com>
* Update creating-custom-workloads.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* updating changes suggested by Naarcha-AWS
Signed-off-by: Samuel Valdes Gutierrez <valdesgutierrez@gmail.com>
* updating changes suggested by Naarcha-AWS
Signed-off-by: Samuel Valdes Gutierrez <valdesgutierrez@gmail.com>
* Rename _benchmark/workloads/index.md to _benchmark/workloads/reference/index.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Rename _benchmark/workloads/indices.md to _benchmark/workloads/reference/indices.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Samuel Valdes Gutierrez <valdesgutierrez@gmail.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-10-31 10:18:45 -05:00
03d177aa2b
Add documentation for Data Prepper translate processor ( #5122 )
...
* Add translate processor docs
Signed-off-by: Hai Yan <oeyh@amazon.com>
* Fix a table display issue
Signed-off-by: Hai Yan <oeyh@amazon.com>
* Address review comments: tweak sentences; add type info; add top level options
Signed-off-by: Hai Yan <oeyh@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/processors/translate.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update translate.md
* Update translate.md
* Update _data-prepper/pipelines/configuration/processors/translate.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Hai Yan <oeyh@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-10-12 08:53:20 -05:00
f4984fb605
Documentation for the Data Prepper OpenSearch source ( #5144 )
...
* Documentation for the Data Prepper OpenSearch source
Signed-off-by: Taylor Gray <tylgry@amazon.com>
* Update _data-prepper/pipelines/configuration/sources/opensearch.md
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Taylor Gray <graytaylor0@gmail.com>
* Update opensearch.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/sources/opensearch.md
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/sources/opensearch.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update opensearch.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/sources/opensearch.md
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Taylor Gray <graytaylor0@gmail.com>
* Update _data-prepper/pipelines/configuration/sources/opensearch.md
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Taylor Gray <graytaylor0@gmail.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Taylor Gray <tylgry@amazon.com>
Signed-off-by: Taylor Gray <graytaylor0@gmail.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-10-11 13:48:53 -05:00
55fb97b290
Update Data Prepper key_value processor docs ( #5153 )
...
* Add new options to table
Signed-off-by: Hai Yan <oeyh@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/processors/key-value.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/processors/key-value.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Hai Yan <oeyh@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-10-11 10:00:57 -05:00
66db871345
Add docs for Data Prepper dissect processor ( #5159 )
...
* Add docs for dissect processor
Signed-off-by: Hai Yan <oeyh@amazon.com>
* Fix some style issues
Signed-off-by: Hai Yan <oeyh@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/processors/dissect.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Hai Yan <oeyh@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-10-11 10:00:28 -05:00
7eceb2baf0
Update documentation for new AD settings ( #4835 )
...
* Update documentation for new AD settings
Signed-off-by: Jonah Calvo <caljonah@amazon.com>
* update wording for verbose
Signed-off-by: Jonah Calvo <caljonah@amazon.com>
* Update _data-prepper/pipelines/configuration/processors/anomaly-detector.md
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Jonah Calvo <jonah.calvo@gmail.com>
* Update _data-prepper/pipelines/configuration/processors/anomaly-detector.md
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Jonah Calvo <jonah.calvo@gmail.com>
* Update _data-prepper/pipelines/configuration/processors/anomaly-detector.md
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Jonah Calvo <jonah.calvo@gmail.com>
* Remove 'few' from description
Signed-off-by: Jonah Calvo <caljonah@amazon.com>
---------
Signed-off-by: Jonah Calvo <caljonah@amazon.com>
Signed-off-by: Jonah Calvo <jonah.calvo@gmail.com>
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
2023-09-21 15:07:38 -05:00
0e2a5888a9
Typo in convert_entry_type.md ( #4994 )
...
Replaced "convert_entry_type_type" with "convert_entry_type".
Signed-off-by: Vlad Babak <129367331+vlad-voxalyze@users.noreply.github.com>
2023-09-08 14:03:14 -05:00
ea886a4763
Fix S3 bug ( #4974 )
...
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-09-06 16:01:03 -05:00
64d59b9bb2
Behavioral changes for Data Prepper S3 sink ( #4897 )
...
* Updates the Data Prepper documentation for S3 sinks based on recent behavior changes.
Signed-off-by: David Venable <dlv@amazon.com>
* Updates from the PR feedback.
Signed-off-by: David Venable <dlv@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-08-29 10:01:24 -05:00
27cbf9d03a
Removes the acknowledgements_timeout configuration from the Data Prepper Kafka source. ( #4898 )
...
Signed-off-by: David Venable <dlv@amazon.com>
2023-08-28 11:47:36 -05:00
000aecc1a3
Moves the generic sink options into the Sinks documentation rather than PIpeline Configuration options. ( #4907 )
...
Signed-off-by: David Venable <dlv@amazon.com>
2023-08-28 10:43:26 -05:00
5306134c41
Updated the intro text for the s3 sink. ( #4854 )
...
Signed-off-by: David Venable <dlv@amazon.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-08-23 13:33:46 -05:00
1250649984
Fix typos on S3 sink documentation ( #4862 )
...
* Fix typos on S3 sink documentation
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/sinks/s3.md
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
2023-08-23 12:49:46 -05:00
ed21ec74cf
Removes the Data Prepper csv sink codec. This is not ready for use and should not be documented. ( #4859 )
...
Signed-off-by: David Venable <dlv@amazon.com>
2023-08-23 09:19:04 -05:00
cc21df7ee8
S3 scan doc update ( #4848 )
...
* Updated s3 scan docs
Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-08-22 15:28:13 -05:00
0b5011409a
Data Prepper documentation updates for the s3 sink in 2.4.0 ( #4847 )
...
* Data Prepper documentation updates for the s3 sink in 2.4.0. Includes other corrections.
* Updated the Parquet/Avro codecs to include auto_schema as well as some clarifications on the restrictions of the auto-generated schema.
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-08-22 15:18:40 -05:00
8e9d0ed4de
S3 scan docs ( #4756 )
...
* Add notification_source option for S3
Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>
* Added S3 scan configuration
Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>
* Addressed some failing style checks
Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>
* Edit S3 additions.
* Fix links.
* Fix style flaw
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update s3.md
---------
Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-08-21 11:02:07 -05:00
ac3f60d5ca
Documentation for bucket ownership features in Data Prepper 2.4 ( #4679 )
...
* Adds documentation for new features in Data Prepper 2.4 related to bucket ownership. Includes a section explaining how the bucket ownership works and can be configured for cross-account access. Resolves #4678
Signed-off-by: David Venable <dlv@amazon.com>
* Updates from PR feedback.
Signed-off-by: David Venable <dlv@amazon.com>
* Remove redundant header
Add a name to primary H2 header to section to prevent links from breaking.
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-08-21 10:46:04 -05:00
7d077786bc
MAINT: update distribution_version in opensearch sink ( #4742 )
...
* MAINT: update distribution_version in opensearch sink
Signed-off-by: George Chen <qchea@amazon.com>
* Update _data-prepper/pipelines/configuration/sinks/opensearch.md
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Qi Chen <chenqi0805@gmail.com>
* Update _data-prepper/pipelines/configuration/sinks/opensearch.md
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Qi Chen <chenqi0805@gmail.com>
* Fix typo
* ADD: request compression flag
Signed-off-by: George Chen <qchea@amazon.com>
* Update opensearch.md
---------
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Qi Chen <chenqi0805@gmail.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-08-21 10:45:40 -05:00
152ffcd79f
Add Kafka source for DP 2.4 ( #4737 )
...
* Add Kafka source for DP 2.4
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Use new Kafka settings
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Add technical feedback
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Fix typos
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Apply suggestions from code review
Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM>
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Change array to object.
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Last minute changes to Kafka defaults
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-08-21 10:43:32 -05:00
09ff9fc10c
Add documentation for managing data prepper source coordination ( #4758 )
...
* Add documentation for managing data prepper source coordination
Signed-off-by: Taylor Gray <tylgry@amazon.com>
* Update source-coordination.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/managing-data-prepper/source-coordination.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Add TTL link
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Taylor Gray <tylgry@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-08-21 10:39:50 -05:00
442f4c446a
Documents general sink options ( #4759 )
...
* Adds general sink options which includes: routes, tags_target_key, include_keys, exclude_keys.
Signed-off-by: David Venable <dlv@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-08-21 10:39:31 -05:00
c14b8b2a07
Add CIDR and User Agent processors ( #4585 )
...
* Add CIDR and User Agent processors
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Add CIDR functions
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Fix typos
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Add technical feedback
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Update user-agent.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/expression-syntax.md
Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update user-agent.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-07-27 14:58:47 -05:00
0fcfc4862e
Add S3 sink documentation ( #4340 )
...
* Add S3 sink documentation
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Update _data-prepper/pipelines/configuration/sinks/s3.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/sinks/s3.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Update _data-prepper/pipelines/configuration/sinks/s3.md
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-07-18 16:06:06 -07:00
adae164d53
Add obfuscate processor ( #4515 )
...
* Add obfuscate processor reference
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* fix typo
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Fix paragraph description.
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Fix consistency issues
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
* Apply suggestions from code review
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-07-14 12:44:01 -05:00
d78bd5b70e
Update capitalization for Events ( #4213 )
...
Signed-off-by: Powershellfish <47839113+powershellfish@users.noreply.github.com>
2023-07-05 13:50:27 -05:00
5a4daa4225
Documents the Data Prepper opensearch sink's template_type parameter. ( #4290 )
...
* Documents the Data Prepper opensearch sink's template_type parameter.
Signed-off-by: David Venable <dlv@amazon.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
* Apply suggestions from code review
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
---------
Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-06-13 13:53:29 -05:00
3b9ae603de
Updated capitalization. ( #4214 )
...
Signed-off-by: Powershellfish <47839113+powershellfish@users.noreply.github.com>
2023-06-07 06:41:42 -07:00
91ca0ffd55
redirect ( #4198 )
...
Signed-off-by: Heather Halter <hdhalter@amazon.com>
2023-05-26 15:57:18 -07:00
baa2fb45ef
[DOC] Remove indentation in copy_values processor ( #4057 )
...
* Test.
Signed-off-by: Powershellfish <47839113+powershellfish@users.noreply.github.com>
* Update index.md
Testing ability to create PRs.
* Removed indentation.
Signed-off-by: Powershellfish <47839113+powershellfish@users.noreply.github.com>
---------
Signed-off-by: Powershellfish <47839113+powershellfish@users.noreply.github.com>
2023-05-15 17:49:13 -07:00
Melissa Vagi
Travis Benedict
Qi Chen
kolchfa-aws
kkondaka
David Venable
Craig Adam
Naarcha-AWS
MUHAMMAD SAMIULLAH
Qi Chen
Samuel Valdes Gutierrez
Hai Yan
Taylor Gray
Jonah Calvo
Vlad Babak
Asif Sohail Mohammed
Powershellfish
Heather Halter