Commit Graph

10 Commits

Author SHA1 Message Date
David Venable 97d14ccfc2
Data Prepper's s3 source: visibility_duplication_protection (#5665)
* Adds documentation for Data Prepper's s3 source visibility_duplication_protection configuration. Includes the necessary permissions.

Signed-off-by: David Venable <dlv@amazon.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

---------

Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-11-27 11:26:54 -06:00
Naarcha-AWS 72b3363460
Add codec and processor combinations to S3 source (#5288)
* Add codec and processor combinations to S3 source.

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Update s3.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Make codec-processor combinations its own file

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Make Data Prepper codec page

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Fix links

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Fix links

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Reformat article without tables.

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* fix link

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/common-use-cases/codec-processor-combinations.md

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update codec-processor-combinations.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

---------

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
2023-11-17 20:06:04 -06:00
Naarcha-AWS ea886a4763
Fix S3 bug (#4974)
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-09-06 16:01:03 -05:00
Asif Sohail Mohammed cc21df7ee8
S3 scan doc update (#4848)
* Updated s3 scan docs

Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

---------

Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
2023-08-22 15:28:13 -05:00
Asif Sohail Mohammed 8e9d0ed4de
S3 scan docs (#4756)
* Add notification_source option for S3

Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

* Added S3 scan configuration

Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

* Addressed some failing style checks

Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

* Edit S3 additions.

* Fix links.

* Fix style flaw

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update s3.md

---------

Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-08-21 11:02:07 -05:00
David Venable ac3f60d5ca
Documentation for bucket ownership features in Data Prepper 2.4 (#4679)
* Adds documentation for new features in Data Prepper 2.4 related to bucket ownership. Includes a section explaining how the bucket ownership works and can be configured for cross-account access. Resolves #4678

Signed-off-by: David Venable <dlv@amazon.com>

* Updates from PR feedback.

Signed-off-by: David Venable <dlv@amazon.com>

* Remove redundant header

Add a name to primary H2 header to section to prevent links from breaking.

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

---------

Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-08-21 10:46:04 -05:00
David Venable aa2033c863
Updates the Data Prepper S3 source documentation (#3813)
* Updates the Data Prepper S3 source documentation with additional instructions for usage and new features in 2.2. Includes some clean-up as well.

Signed-off-by: David Venable <dlv@amazon.com>

* Add back in example

* Update _data-prepper/pipelines/configuration/sources/s3.md

Co-authored-by: Caroline <113052567+carolxob@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Co-authored-by: Caroline <113052567+carolxob@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Co-authored-by: Caroline <113052567+carolxob@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Co-authored-by: Caroline <113052567+carolxob@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update s3.md

* Apply suggestions from code review

Co-authored-by: Caroline <113052567+carolxob@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update s3.md

Implement Caroline's feedback

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

---------

Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Caroline <113052567+carolxob@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-04-25 13:10:40 -05:00
Naarcha-AWS 098f6d961a
Add E2E acknowledgements (#3811)
* Add E2E acknowledgements

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Fix typos

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Remove E from achknowledgments

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* another e

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Fix link

* Adjust S3 source.

* Update _data-prepper/pipelines/configuration/sources/s3.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

* Update _data-prepper/pipelines/configuration/sources/s3.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Update _data-prepper/pipelines/pipelines.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Update _data-prepper/pipelines/pipelines.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Update _data-prepper/pipelines/pipelines.md

---------

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-04-20 12:39:24 -05:00
Caroline ef83f6d7d5
Added metrics sections to Aggregate processor subpages. (#2730)
* Added metrics section to Aggregate processor page.

Signed-off-by: carolxob <carolxob@amazon.com>

* Added Metrics section to individual Processors pages.

Signed-off-by: carolxob <carolxob@amazon.com>

* Added metrics section for JSON processor.

Signed-off-by: carolxob <carolxob@amazon.com>

* Added metrics sections. Changed Default is to Default value is.

Signed-off-by: carolxob <carolxob@amazon.com>

* Corrected references from AWS S3 to Amazon S3.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor updates to Metrics sections and phrasing.

Signed-off-by: carolxob <carolxob@amazon.com>

* Updated Action link.

Signed-off-by: carolxob <carolxob@amazon.com>

* Updates based on tech review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Updates based on tech review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Tech review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor updates to buffer_size and batch_size default values.

Signed-off-by: carolxob <carolxob@amazon.com>

* Edits to Metrics sections for each processor.

Signed-off-by: carolxob <carolxob@amazon.com>

* Update made based ondoc review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor updates to intro text for processor pages. Minor adjustements to other text for clarity.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edits.

Signed-off-by: carolxob <carolxob@amazon.com>

* Adjustements to phrasing, fixed typos.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor updates to word choice and corrected a typo.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edit.

Signed-off-by: carolxob <carolxob@amazon.com>

* Made updates based ondoc review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Updates to http-source.

Signed-off-by: carolxob <carolxob@amazon.com>

* Added common processors table to affected docs.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor update to one file.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor update based on tech review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edits.

Signed-off-by: carolxob <carolxob@amazon.com>

* Major editorial feedback incorporated through key-value.md.

Signed-off-by: carolxob <carolxob@amazon.com>

* Incorporated major editorial feedback thup to service-map-stateful.

Signed-off-by: carolxob <carolxob@amazon.com>

* Incorporated major editorial feedback for Processors section.

Signed-off-by: carolxob <carolxob@amazon.com>

* Major editorial updates, specifically to inclusion of text introducing option configuration tables.

Signed-off-by: carolxob <carolxob@amazon.com>

* Major editorial feedback through otel-trace.md incorporated.

Signed-off-by: carolxob <carolxob@amazon.com>

* Major editorial edits incorporated.

Signed-off-by: carolxob <carolxob@amazon.com>

* Technical feedback and editorial feedback incorporated.

Signed-off-by: carolxob <carolxob@amazon.com>

* Incorporated missing editorial feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor adjustements to OpenSearch sink.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor changes to capitalization.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edits.

Signed-off-by: carolxob <carolxob@amazon.com>

* Made one instance of processor name consistent with other references.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor update based on editorial feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

---------

Signed-off-by: carolxob <carolxob@amazon.com>
2023-02-27 11:40:59 -05:00
Caroline 0249991f76
Data Prepper ToC Update (#2514)
* Creating PR with first file.

Signed-off-by: carolxob <carolxob@amazon.com>

* Adding newly created files to PR.

Signed-off-by: carolxob <carolxob@amazon.com>

* Reorganized files and added appropriate metadata to map ToC correctly.

Signed-off-by: carolxob <carolxob@amazon.com>

* Moved Authoring pipelines page.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor ToC updates.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor ToC updates to Sources section for Data Prepper.

Signed-off-by: carolxob <carolxob@amazon.com>

* Updated Buffers section under Data Prepper.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor update to otelmetricssource.

Signed-off-by: carolxob <carolxob@amazon.com>

* Restructured ToC in Processors section for Data Prepper.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor filename change.

Signed-off-by: carolxob <carolxob@amazon.com>

* Adjustments to metadata in ToC.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edit.

Signed-off-by: carolxob <carolxob@amazon.com>

* Fixed nav order in metadata.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edit.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor update top metadata for ToC.

Signed-off-by: carolxob <carolxob@amazon.com>

* Adjustmenets to Toc order.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor adjustments to Sinks section.

Signed-off-by: carolxob <carolxob@amazon.com>

* Adjustements to high level ToC.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor adjustement to Pipelines.md

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor update.

Signed-off-by: carolxob <carolxob@amazon.com>

* Slight reorganization. Removed two placeholder pages for now.

Signed-off-by: carolxob <carolxob@amazon.com>

* Removed a page and replaced with pipelines content.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor changes/additions to content for placeholder pages.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor update to page link.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edits.

Signed-off-by: carolxob <carolxob@amazon.com>

* Removed /clients from redirects to correct nav order.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edits.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor adjustments.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor adjustment ot metadata.

Signed-off-by: carolxob <carolxob@amazon.com>

* TOC link fixes

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

* Changed page name.

Signed-off-by: carolxob <carolxob@amazon.com>

* Corrected references to Peer Forwarder.

Signed-off-by: carolxob <carolxob@amazon.com>

* Renamed Data Prepper folder.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor updates to phrasing and capitalization.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor phrasing update.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor phrasing update.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor change.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor change to change S3 Source to S3Source.

Signed-off-by: carolxob <carolxob@amazon.com>

* Updated references to peer forwarder and changed capitalization.

Signed-off-by: carolxob <carolxob@amazon.com>

* Updated capitalization for peer forwarder.

Signed-off-by: carolxob <carolxob@amazon.com>

* Made edits based on doc review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Update to one word.

Signed-off-by: carolxob <carolxob@amazon.com>

---------

Signed-off-by: carolxob <carolxob@amazon.com>
Signed-off-by: Naarcha-AWS <naarcha@amazon.com>
Co-authored-by: Naarcha-AWS <naarcha@amazon.com>
2023-02-03 15:06:10 -07:00