More Data Prepper changes (#3297)

* more DP changes

Signed-off-by: Heather Halter <hdhalter@amazon.com>

* updates from doc

Signed-off-by: Heather Halter <hdhalter@amazon.com>

---------

Signed-off-by: Heather Halter <hdhalter@amazon.com>
This commit is contained in:
Heather Halter 2023-03-08 14:55:47 -08:00 committed by GitHub
parent 1c34edb2e3
commit f01567de44
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 15 additions and 9 deletions

View File

@ -10,18 +10,19 @@ redirect_from:
Data Prepper is an independent component, not an OpenSearch plugin, that converts data for use with OpenSearch. It's not bundled with the all-in-one OpenSearch installation packages.
If you are migrating from Open Distro Data Prepper, see the [Migrating from Open Distro]({{site.url}}{{site.baseurl}}/data-prepper/migrate-open-distro/) page.
If you are migrating from Open Distro Data Prepper, see [Migrating from Open Distro]({{site.url}}{{site.baseurl}}/data-prepper/migrate-open-distro/).
{: .note}
## 1. Installing Data Prepper
There are two ways to install Data Prepper: Run the Docker image and build from source.
There are two ways to install Data Prepper: you can run the Docker image or build from source.
The easiest way to use Data Prepper is by running the Docker image. We suggest that you use this approach if you have [Docker](https://www.docker.com) available. Do the following:
The easiest way to use Data Prepper is by running the Docker image. We suggest that you use this approach if you have [Docker](https://www.docker.com) available. Run the following command:
```
docker pull opensearchproject/data-prepper:latest
```
{% include copy.html %}
If you have special requirements that require you to build from source, or if you want to contribute, see the [Developer Guide](https://github.com/opensearch-project/data-prepper/blob/main/docs/developer_guide.md).
@ -50,6 +51,7 @@ simple-sample-pipeline:
sink:
- stdout:
```
{% include copy.html %}
## 4. Running Data Prepper
@ -61,6 +63,7 @@ docker run --name data-prepper \
opensearchproject/data-prepper:latest
```
{% include copy.html %}
The example pipeline configuration above demonstrates a simple pipeline with a source (`random`) sending data to a sink (`stdout`). For examples of more advanced pipeline configurations, see [Pipelines]({{site.url}}{{site.baseurl}}/clients/data-prepper/pipelines/).
@ -93,24 +96,27 @@ For Data Prepper 2.0 or later, use this command:
```
docker run --name data-prepper -p 4900:4900 -v ${PWD}/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml -v ${PWD}/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml opensearchproject/data-prepper:latest
```
{% include copy.html %}
For Data Prepper 2.0 and earlier, use this command:
For Data Prepper versions earlier than 2.0, use this command:
```
docker run --name data-prepper -p 4900:4900 -v ${PWD}/pipelines.yaml:/usr/share/data-prepper/pipelines.yaml -v ${PWD}/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml opensearchproject/data-prepper:1.x
```
{% include copy.html %}
Once Data Prepper is running, it processes data until it is shut down. Once you are done, shut it down with the following command:
```
curl -X POST http://localhost:4900/shutdown
POST /shutdown
```
{% include copy-curl.html %}
### Additional configurations
For Data Prepper 2.0 or later, the Log4j 2 configuration file is read from `config/log4j2.properties` in the application's home directory. By default, it uses `log4j2-rolling.properties` in the *shared-config* directory.
For Data Prepper 1.5 or earlier, optionally add `"-Dlog4j.configurationFile=config/log4j2.properties"` to the command if you would like to pass a custom log4j2 properties file. If no properties file is provided, Data Prepper defaults to the log4j2.properties file in the *shared-config* directory.
For Data Prepper 1.5 or earlier, optionally add `"-Dlog4j.configurationFile=config/log4j2.properties"` to the command if you want to pass a custom log4j2 properties file. If no properties file is provided, Data Prepper defaults to the log4j2.properties file in the *shared-config* directory.
## Next steps
@ -120,8 +126,8 @@ Log ingestion is also an important Data Prepper use case. To learn more, see [Lo
To learn how to run Data Prepper with a Logstash configuration, see [Migrating from Logstash]({{site.url}}{{site.baseurl}}/data-prepper/migrating-from-logstash-data-prepper/).
For information on how to monitor Data Prepper, see the [Monitoring]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/monitoring/) page.
For information on how to monitor Data Prepper, see [Monitoring]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/monitoring/).
## Other examples
## More examples
We have several other Docker [examples](https://github.com/opensearch-project/data-prepper/tree/main/examples/) that allow you to run Data Prepper in different scenarios.
For more examples of Data Prepper, see [examples](https://github.com/opensearch-project/data-prepper/tree/main/examples/) in the Data Prepper repo.