Expand configuration information in Data Prepper Getting started guide. (#3298)
* Expand configuration information in Data Prepper Getting started guide. Signed-off-by: carolxob <carolxob@amazon.com> * Minor copy edits. Signed-off-by: carolxob <carolxob@amazon.com> * Copy edits for the Configuration section. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Update _data-prepper/getting-started.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> * Update _data-prepper/getting-started.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> * Update _data-prepper/getting-started.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> * Update _data-prepper/getting-started.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> * Update _data-prepper/getting-started.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> * Update _data-prepper/getting-started.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> * Update _data-prepper/getting-started.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> * Update _data-prepper/getting-started.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> * Update _data-prepper/getting-started.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> * Made edits based on doc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Made edits based on doc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Update _data-prepper/getting-started.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Update _data-prepper/getting-started.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Update _data-prepper/getting-started.md Co-authored-by: Nathan Bower <nbower@amazon.com> * Update _data-prepper/getting-started.md Co-authored-by: Nathan Bower <nbower@amazon.com> --------- Signed-off-by: carolxob <carolxob@amazon.com> Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
This commit is contained in:
Caroline
GitHub
parent
6335a197f9
commit
f154992ccd
|
|
@ -28,10 +28,37 @@ If you have special requirements that require you to build from source, or if yo
|
|||
|
||||
## 2. Configuring Data Prepper
|
||||
|
||||
You must configure Data Prepper with a pipeline before running it. You'll modify the following files:
|
||||
Two configuration files are required to run a Data Prepper instance. Optionally, you can configure a Log4j 2 configuration file. See [Configuring Log4j]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-log4j/) for more information. The following list describes the purpose of each configuration file:
|
||||
|
||||
* `data-prepper-config.yaml`
|
||||
* `pipelines.yaml`
|
||||
* `pipelines.yaml`: This file describes which data pipelines to run, including sources, processors, and sinks.
|
||||
* `data-prepper-config.yaml`: This file contains Data Prepper server settings that allow you to interact with exposed Data Prepper server APIs.
|
||||
* `log4j2-rolling.properties` (optional): This file contains Log4j 2 configuration options and can be a JSON, YAML, XML, or .properties file type.
|
||||
|
||||
For Data Prepper versions earlier than 2.0, the `.jar` file expects the pipeline configuration file path to be followed by the server configuration file path. See the following configuration path example:
|
||||
|
||||
```
|
||||
java -jar data-prepper-core-$VERSION.jar pipelines.yaml data-prepper-config.yaml
|
||||
```
|
||||
|
||||
Optionally, you can add `"-Dlog4j.configurationFile=config/log4j2.properties"` to the command to pass a custom Log4j 2 configuration file. If you don't provide a properties file, Data Prepper defaults to the `log4j2.properties` file in the `shared-config` directory.
|
||||
|
||||
|
||||
Starting with Data Prepper 2.0, you can launch Data Prepper by using the following `data-prepper` script that does not require any additional command line arguments:
|
||||
|
||||
```
|
||||
bin/data-prepper
|
||||
```
|
||||
|
||||
Configuration files are read from specific subdirectories in the application's home directory:
|
||||
1. `pipelines/`: Used for pipeline configurations. Pipeline configurations can be written in one or more YAML files.
|
||||
2. `config/data-prepper-config.yaml`: Used for the Data Prepper server configuration.
|
||||
|
||||
You can supply your own pipeline configuration file path followed by the server configuration file path. However, this method will not be supported in a future release. See the following example:
|
||||
```
|
||||
bin/data-prepper pipelines.yaml data-prepper-config.yaml
|
||||
```
|
||||
|
||||
The Log4j 2 configuration file is read from the `config/log4j2.properties` file located in the application's home directory.
|
||||
|
||||
To configure Data Prepper, see the following information for each use case:
|
||||
|
||||
|
|
@ -40,7 +67,7 @@ To configure Data Prepper, see the following information for each use case:
|
|||
|
||||
## 3. Defining a pipeline
|
||||
|
||||
Create a Data Prepper pipeline file, `pipelines.yaml`, with the following configuration:
|
||||
Create a Data Prepper pipeline file named `pipelines.yaml` using the following configuration:
|
||||
|
||||
```yml
|
||||
simple-sample-pipeline:
|
||||
|
|
|
|||
Loading…
Reference in New Issue