Expand configuration information in Data Prepper Getting started guide. (#3298)

* Expand configuration information in Data Prepper Getting started guide.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor copy edits.

Signed-off-by: carolxob <carolxob@amazon.com>

* Copy edits for the Configuration section.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edits.

Signed-off-by: carolxob <carolxob@amazon.com>

* Minor edits.

Signed-off-by: carolxob <carolxob@amazon.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com>

* Made edits based on doc review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Made edits based on doc review feedback.

Signed-off-by: carolxob <carolxob@amazon.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

* Update _data-prepper/getting-started.md

Co-authored-by: Nathan Bower <nbower@amazon.com>

---------

Signed-off-by: carolxob <carolxob@amazon.com>
Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
This commit is contained in:
Caroline 2023-03-09 14:22:42 -05:00 committed by GitHub
parent 6335a197f9
commit f154992ccd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 31 additions and 4 deletions

View File

@ -28,10 +28,37 @@ If you have special requirements that require you to build from source, or if yo
## 2. Configuring Data Prepper ## 2. Configuring Data Prepper
You must configure Data Prepper with a pipeline before running it. You'll modify the following files: Two configuration files are required to run a Data Prepper instance. Optionally, you can configure a Log4j 2 configuration file. See [Configuring Log4j]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-log4j/) for more information. The following list describes the purpose of each configuration file:
* `data-prepper-config.yaml` * `pipelines.yaml`: This file describes which data pipelines to run, including sources, processors, and sinks.
* `pipelines.yaml` * `data-prepper-config.yaml`: This file contains Data Prepper server settings that allow you to interact with exposed Data Prepper server APIs.
* `log4j2-rolling.properties` (optional): This file contains Log4j 2 configuration options and can be a JSON, YAML, XML, or .properties file type.
For Data Prepper versions earlier than 2.0, the `.jar` file expects the pipeline configuration file path to be followed by the server configuration file path. See the following configuration path example:
```
java -jar data-prepper-core-$VERSION.jar pipelines.yaml data-prepper-config.yaml
```
Optionally, you can add `"-Dlog4j.configurationFile=config/log4j2.properties"` to the command to pass a custom Log4j 2 configuration file. If you don't provide a properties file, Data Prepper defaults to the `log4j2.properties` file in the `shared-config` directory.
Starting with Data Prepper 2.0, you can launch Data Prepper by using the following `data-prepper` script that does not require any additional command line arguments:
```
bin/data-prepper
```
Configuration files are read from specific subdirectories in the application's home directory:
1. `pipelines/`: Used for pipeline configurations. Pipeline configurations can be written in one or more YAML files.
2. `config/data-prepper-config.yaml`: Used for the Data Prepper server configuration.
You can supply your own pipeline configuration file path followed by the server configuration file path. However, this method will not be supported in a future release. See the following example:
```
bin/data-prepper pipelines.yaml data-prepper-config.yaml
```
The Log4j 2 configuration file is read from the `config/log4j2.properties` file located in the application's home directory.
To configure Data Prepper, see the following information for each use case: To configure Data Prepper, see the following information for each use case:
@ -40,7 +67,7 @@ To configure Data Prepper, see the following information for each use case:
## 3. Defining a pipeline ## 3. Defining a pipeline
Create a Data Prepper pipeline file, `pipelines.yaml`, with the following configuration: Create a Data Prepper pipeline file named `pipelines.yaml` using the following configuration:
```yml ```yml
simple-sample-pipeline: simple-sample-pipeline: