opensearch-docs-cn/_clients/data-prepper/get-started.md

64 lines
2.7 KiB
Markdown
Raw Normal View History

---
layout: default
title: Get Started
parent: Data Prepper
nav_order: 1
---
# Get started with Data Prepper
Data Prepper is an independent component, not an OpenSearch plugin, that converts data for use with OpenSearch. It's not bundled with the all-in-one OpenSearch installation packages.
## 1. Install Data Prepper
To use the Docker image, pull it like any other image:
```bash
docker pull opensearchproject/data-prepper:latest
```
## 2. Define a pipeline
Data Prepper 2.0 documentation (#1510) * Change Data Prepper intro Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Add next steps section Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Add David's feedback Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Fix optional tags Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Address small typo * [Data Prepper 2.0]MAINT: documentation change regarding record type (#1306) * MAINT: documentation change regarding record type Signed-off-by: Chen <qchea@amazon.com> * MAINT: documentation on trace group fields Signed-off-by: Chen <qchea@amazon.com> Signed-off-by: Chen <qchea@amazon.com> * Update docs for Data Prepper 2.0 (#1404) * Update get-started Signed-off-by: Hai Yan <oeyh@amazon.com> * Update pipelines.md Signed-off-by: Hai Yan <oeyh@amazon.com> * Add peer forwarder options to references Signed-off-by: Hai Yan <oeyh@amazon.com> * Add csv processor options to refereces Signed-off-by: Hai Yan <oeyh@amazon.com> * Add docs for conditional routing Signed-off-by: Hai Yan <oeyh@amazon.com> * Add docs for json processor Signed-off-by: Hai Yan <oeyh@amazon.com> * Remove docs for peer forwarder plugin Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review feedback - revise sentences, fix inaccurate info and typos Signed-off-by: Hai Yan <oeyh@amazon.com> * Add missing options for http source and peer forwarder Signed-off-by: Hai Yan <oeyh@amazon.com> * Update ssl options on peer forwarder Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: Hai Yan <oeyh@amazon.com> * More updates for Data Prepper 2.0 (#1469) * Update http source and opensearch sink options Signed-off-by: Hai Yan <oeyh@amazon.com> * Update docker run command Signed-off-by: Hai Yan <oeyh@amazon.com> * Add more missing options Signed-off-by: Hai Yan <oeyh@amazon.com> * Add metadata_root_key for s3 source Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments - tweak sentences and fix typos Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: Hai Yan <oeyh@amazon.com> * Fix broken link Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Signed-off-by: Chen <qchea@amazon.com> Signed-off-by: Hai Yan <oeyh@amazon.com> Co-authored-by: Naarcha-AWS <naarcha@amazon.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Qi Chen <qchea@amazon.com> Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>
2022-10-11 12:36:37 -04:00
Create a Data Prepper pipeline file, `my-pipelines.yaml`, with the following configuration:
```yml
simple-sample-pipeline:
workers: 2
delay: "5000"
source:
random:
sink:
- stdout:
```
## 3. Start Data Prepper
Run the following command with your pipeline configuration YAML.
```bash
docker run --name data-prepper \
Data Prepper 2.0 documentation (#1510) * Change Data Prepper intro Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Add next steps section Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Add David's feedback Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Fix optional tags Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Address small typo * [Data Prepper 2.0]MAINT: documentation change regarding record type (#1306) * MAINT: documentation change regarding record type Signed-off-by: Chen <qchea@amazon.com> * MAINT: documentation on trace group fields Signed-off-by: Chen <qchea@amazon.com> Signed-off-by: Chen <qchea@amazon.com> * Update docs for Data Prepper 2.0 (#1404) * Update get-started Signed-off-by: Hai Yan <oeyh@amazon.com> * Update pipelines.md Signed-off-by: Hai Yan <oeyh@amazon.com> * Add peer forwarder options to references Signed-off-by: Hai Yan <oeyh@amazon.com> * Add csv processor options to refereces Signed-off-by: Hai Yan <oeyh@amazon.com> * Add docs for conditional routing Signed-off-by: Hai Yan <oeyh@amazon.com> * Add docs for json processor Signed-off-by: Hai Yan <oeyh@amazon.com> * Remove docs for peer forwarder plugin Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review feedback - revise sentences, fix inaccurate info and typos Signed-off-by: Hai Yan <oeyh@amazon.com> * Add missing options for http source and peer forwarder Signed-off-by: Hai Yan <oeyh@amazon.com> * Update ssl options on peer forwarder Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: Hai Yan <oeyh@amazon.com> * More updates for Data Prepper 2.0 (#1469) * Update http source and opensearch sink options Signed-off-by: Hai Yan <oeyh@amazon.com> * Update docker run command Signed-off-by: Hai Yan <oeyh@amazon.com> * Add more missing options Signed-off-by: Hai Yan <oeyh@amazon.com> * Add metadata_root_key for s3 source Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments - tweak sentences and fix typos Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: Hai Yan <oeyh@amazon.com> * Fix broken link Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Signed-off-by: Chen <qchea@amazon.com> Signed-off-by: Hai Yan <oeyh@amazon.com> Co-authored-by: Naarcha-AWS <naarcha@amazon.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Qi Chen <qchea@amazon.com> Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>
2022-10-11 12:36:37 -04:00
-v /full/path/to/my-pipelines.yaml:/usr/share/data-prepper/pipelines/my-pipelines.yaml \
opensearchproject/opensearch-data-prepper:latest
```
This sample pipeline configuration above demonstrates a simple pipeline with a source (`random`) sending data to a sink (`stdout`). For more examples and details on more advanced pipeline configurations, see [Pipelines]({{site.url}}{{site.baseurl}}/clients/data-prepper/pipelines).
After starting Data Prepper, you should see log output and some UUIDs after a few seconds:
Data Prepper 2.0 documentation (#1510) * Change Data Prepper intro Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Add next steps section Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Add David's feedback Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Fix optional tags Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Address small typo * [Data Prepper 2.0]MAINT: documentation change regarding record type (#1306) * MAINT: documentation change regarding record type Signed-off-by: Chen <qchea@amazon.com> * MAINT: documentation on trace group fields Signed-off-by: Chen <qchea@amazon.com> Signed-off-by: Chen <qchea@amazon.com> * Update docs for Data Prepper 2.0 (#1404) * Update get-started Signed-off-by: Hai Yan <oeyh@amazon.com> * Update pipelines.md Signed-off-by: Hai Yan <oeyh@amazon.com> * Add peer forwarder options to references Signed-off-by: Hai Yan <oeyh@amazon.com> * Add csv processor options to refereces Signed-off-by: Hai Yan <oeyh@amazon.com> * Add docs for conditional routing Signed-off-by: Hai Yan <oeyh@amazon.com> * Add docs for json processor Signed-off-by: Hai Yan <oeyh@amazon.com> * Remove docs for peer forwarder plugin Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review feedback - revise sentences, fix inaccurate info and typos Signed-off-by: Hai Yan <oeyh@amazon.com> * Add missing options for http source and peer forwarder Signed-off-by: Hai Yan <oeyh@amazon.com> * Update ssl options on peer forwarder Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: Hai Yan <oeyh@amazon.com> * More updates for Data Prepper 2.0 (#1469) * Update http source and opensearch sink options Signed-off-by: Hai Yan <oeyh@amazon.com> * Update docker run command Signed-off-by: Hai Yan <oeyh@amazon.com> * Add more missing options Signed-off-by: Hai Yan <oeyh@amazon.com> * Add metadata_root_key for s3 source Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments - tweak sentences and fix typos Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: Hai Yan <oeyh@amazon.com> * Fix broken link Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Signed-off-by: Chen <qchea@amazon.com> Signed-off-by: Hai Yan <oeyh@amazon.com> Co-authored-by: Naarcha-AWS <naarcha@amazon.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Qi Chen <qchea@amazon.com> Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>
2022-10-11 12:36:37 -04:00
```
2021-09-30T20:19:44,147 [main] INFO org.opensearch.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server running at :4900
2021-09-30T20:19:44,681 [random-source-pool-0] INFO org.opensearch.dataprepper.plugins.source.RandomStringSource - Writing to buffer
2021-09-30T20:19:45,183 [random-source-pool-0] INFO org.opensearch.dataprepper.plugins.source.RandomStringSource - Writing to buffer
2021-09-30T20:19:45,687 [random-source-pool-0] INFO org.opensearch.dataprepper.plugins.source.RandomStringSource - Writing to buffer
2021-09-30T20:19:46,191 [random-source-pool-0] INFO org.opensearch.dataprepper.plugins.source.RandomStringSource - Writing to buffer
2021-09-30T20:19:46,694 [random-source-pool-0] INFO org.opensearch.dataprepper.plugins.source.RandomStringSource - Writing to buffer
2021-09-30T20:19:47,200 [random-source-pool-0] INFO org.opensearch.dataprepper.plugins.source.RandomStringSource - Writing to buffer
2021-09-30T20:19:49,181 [simple-test-pipeline-processor-worker-1-thread-1] INFO org.opensearch.dataprepper.pipeline.ProcessWorker - simple-test-pipeline Worker: Processing 6 records from buffer
{"message":"1043a78e-1312-4341-8c1e-227e34a1fbf3"}
{"message":"b1529b81-1ee1-4cdb-b5d7-11586e570ae6"}
{"message":"56d83593-4c95-4bc4-9c0b-e061d9b23192"}
{"message":"254153df-4534-4f5e-bb31-98b984f2ac29"}
{"message":"ad1430e6-8486-4d84-a2ef-de30315dea07"}
{"message":"81c5e621-79aa-4850-9bf1-68642d70c1ee"}
```