More Data Prepper changes (#3297)
* more DP changes Signed-off-by: Heather Halter <hdhalter@amazon.com> * updates from doc Signed-off-by: Heather Halter <hdhalter@amazon.com> --------- Signed-off-by: Heather Halter <hdhalter@amazon.com>
This commit is contained in:
parent
1c34edb2e3
commit
f01567de44
|
@ -10,18 +10,19 @@ redirect_from:
|
||||||
|
|
||||||
Data Prepper is an independent component, not an OpenSearch plugin, that converts data for use with OpenSearch. It's not bundled with the all-in-one OpenSearch installation packages.
|
Data Prepper is an independent component, not an OpenSearch plugin, that converts data for use with OpenSearch. It's not bundled with the all-in-one OpenSearch installation packages.
|
||||||
|
|
||||||
If you are migrating from Open Distro Data Prepper, see the [Migrating from Open Distro]({{site.url}}{{site.baseurl}}/data-prepper/migrate-open-distro/) page.
|
If you are migrating from Open Distro Data Prepper, see [Migrating from Open Distro]({{site.url}}{{site.baseurl}}/data-prepper/migrate-open-distro/).
|
||||||
{: .note}
|
{: .note}
|
||||||
|
|
||||||
## 1. Installing Data Prepper
|
## 1. Installing Data Prepper
|
||||||
|
|
||||||
There are two ways to install Data Prepper: Run the Docker image and build from source.
|
There are two ways to install Data Prepper: you can run the Docker image or build from source.
|
||||||
|
|
||||||
The easiest way to use Data Prepper is by running the Docker image. We suggest that you use this approach if you have [Docker](https://www.docker.com) available. Do the following:
|
The easiest way to use Data Prepper is by running the Docker image. We suggest that you use this approach if you have [Docker](https://www.docker.com) available. Run the following command:
|
||||||
|
|
||||||
```
|
```
|
||||||
docker pull opensearchproject/data-prepper:latest
|
docker pull opensearchproject/data-prepper:latest
|
||||||
```
|
```
|
||||||
|
{% include copy.html %}
|
||||||
|
|
||||||
If you have special requirements that require you to build from source, or if you want to contribute, see the [Developer Guide](https://github.com/opensearch-project/data-prepper/blob/main/docs/developer_guide.md).
|
If you have special requirements that require you to build from source, or if you want to contribute, see the [Developer Guide](https://github.com/opensearch-project/data-prepper/blob/main/docs/developer_guide.md).
|
||||||
|
|
||||||
|
@ -50,6 +51,7 @@ simple-sample-pipeline:
|
||||||
sink:
|
sink:
|
||||||
- stdout:
|
- stdout:
|
||||||
```
|
```
|
||||||
|
{% include copy.html %}
|
||||||
|
|
||||||
## 4. Running Data Prepper
|
## 4. Running Data Prepper
|
||||||
|
|
||||||
|
@ -61,6 +63,7 @@ docker run --name data-prepper \
|
||||||
opensearchproject/data-prepper:latest
|
opensearchproject/data-prepper:latest
|
||||||
|
|
||||||
```
|
```
|
||||||
|
{% include copy.html %}
|
||||||
|
|
||||||
The example pipeline configuration above demonstrates a simple pipeline with a source (`random`) sending data to a sink (`stdout`). For examples of more advanced pipeline configurations, see [Pipelines]({{site.url}}{{site.baseurl}}/clients/data-prepper/pipelines/).
|
The example pipeline configuration above demonstrates a simple pipeline with a source (`random`) sending data to a sink (`stdout`). For examples of more advanced pipeline configurations, see [Pipelines]({{site.url}}{{site.baseurl}}/clients/data-prepper/pipelines/).
|
||||||
|
|
||||||
|
@ -93,24 +96,27 @@ For Data Prepper 2.0 or later, use this command:
|
||||||
```
|
```
|
||||||
docker run --name data-prepper -p 4900:4900 -v ${PWD}/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml -v ${PWD}/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml opensearchproject/data-prepper:latest
|
docker run --name data-prepper -p 4900:4900 -v ${PWD}/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml -v ${PWD}/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml opensearchproject/data-prepper:latest
|
||||||
```
|
```
|
||||||
|
{% include copy.html %}
|
||||||
|
|
||||||
For Data Prepper 2.0 and earlier, use this command:
|
For Data Prepper versions earlier than 2.0, use this command:
|
||||||
|
|
||||||
```
|
```
|
||||||
docker run --name data-prepper -p 4900:4900 -v ${PWD}/pipelines.yaml:/usr/share/data-prepper/pipelines.yaml -v ${PWD}/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml opensearchproject/data-prepper:1.x
|
docker run --name data-prepper -p 4900:4900 -v ${PWD}/pipelines.yaml:/usr/share/data-prepper/pipelines.yaml -v ${PWD}/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml opensearchproject/data-prepper:1.x
|
||||||
```
|
```
|
||||||
|
{% include copy.html %}
|
||||||
|
|
||||||
Once Data Prepper is running, it processes data until it is shut down. Once you are done, shut it down with the following command:
|
Once Data Prepper is running, it processes data until it is shut down. Once you are done, shut it down with the following command:
|
||||||
|
|
||||||
```
|
```
|
||||||
curl -X POST http://localhost:4900/shutdown
|
POST /shutdown
|
||||||
```
|
```
|
||||||
|
{% include copy-curl.html %}
|
||||||
|
|
||||||
### Additional configurations
|
### Additional configurations
|
||||||
|
|
||||||
For Data Prepper 2.0 or later, the Log4j 2 configuration file is read from `config/log4j2.properties` in the application's home directory. By default, it uses `log4j2-rolling.properties` in the *shared-config* directory.
|
For Data Prepper 2.0 or later, the Log4j 2 configuration file is read from `config/log4j2.properties` in the application's home directory. By default, it uses `log4j2-rolling.properties` in the *shared-config* directory.
|
||||||
|
|
||||||
For Data Prepper 1.5 or earlier, optionally add `"-Dlog4j.configurationFile=config/log4j2.properties"` to the command if you would like to pass a custom log4j2 properties file. If no properties file is provided, Data Prepper defaults to the log4j2.properties file in the *shared-config* directory.
|
For Data Prepper 1.5 or earlier, optionally add `"-Dlog4j.configurationFile=config/log4j2.properties"` to the command if you want to pass a custom log4j2 properties file. If no properties file is provided, Data Prepper defaults to the log4j2.properties file in the *shared-config* directory.
|
||||||
|
|
||||||
## Next steps
|
## Next steps
|
||||||
|
|
||||||
|
@ -120,8 +126,8 @@ Log ingestion is also an important Data Prepper use case. To learn more, see [Lo
|
||||||
|
|
||||||
To learn how to run Data Prepper with a Logstash configuration, see [Migrating from Logstash]({{site.url}}{{site.baseurl}}/data-prepper/migrating-from-logstash-data-prepper/).
|
To learn how to run Data Prepper with a Logstash configuration, see [Migrating from Logstash]({{site.url}}{{site.baseurl}}/data-prepper/migrating-from-logstash-data-prepper/).
|
||||||
|
|
||||||
For information on how to monitor Data Prepper, see the [Monitoring]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/monitoring/) page.
|
For information on how to monitor Data Prepper, see [Monitoring]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/monitoring/).
|
||||||
|
|
||||||
## Other examples
|
## More examples
|
||||||
|
|
||||||
We have several other Docker [examples](https://github.com/opensearch-project/data-prepper/tree/main/examples/) that allow you to run Data Prepper in different scenarios.
|
For more examples of Data Prepper, see [examples](https://github.com/opensearch-project/data-prepper/tree/main/examples/) in the Data Prepper repo.
|
||||||
|
|
Loading…
Reference in New Issue