opensearch-docs-cn/_data-prepper/managing-data-prepper/core-apis.md

86 lines
3.9 KiB
Markdown
Raw Normal View History

Add Core APIs to doc website repo. (#1932) * Adding Core API doc to doc website repo. Signed-off-by: carolxob <carolxob@amazon.com> * Made edits based on doc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Update _data-prepper/core-apis.md Co-authored-by: Alice Williams <88908598+alicejw-aws@users.noreply.github.com> * Minor edits based on doc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Pushing first set of updates because last ones did not commit. Signed-off-by: carolxob <carolxob@amazon.com> * Made more changes as part of editorial review. Signed-off-by: carolxob <carolxob@amazon.com> * Added more editorial feedback edits. Signed-off-by: carolxob <carolxob@amazon.com> * Pushing more editorial feedback changes. Signed-off-by: carolxob <carolxob@amazon.com> * Added more editorial feedback changes. Signed-off-by: carolxob <carolxob@amazon.com> * Added more editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Made slight changes. Signed-off-by: carolxob <carolxob@amazon.com> * Picked up a few more changes from editorial. Signed-off-by: carolxob <carolxob@amazon.com> * Incorporated doc review feedback. Needs a couple of tech questions addressed. Signed-off-by: carolxob <carolxob@amazon.com> * Minor changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits made from editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor capitalization update. Signed-off-by: carolxob <carolxob@amazon.com> * Update _data-prepper/core-apis.md * Update _data-prepper/core-apis.md Co-authored-by: Nate Bower <nbower@amazon.com> * Made changes based on editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> Signed-off-by: carolxob <carolxob@amazon.com> Co-authored-by: Alice Williams <88908598+alicejw-aws@users.noreply.github.com> Co-authored-by: Nate Bower <nbower@amazon.com>
2023-01-13 13:52:24 -05:00
---
layout: default
title: Core APIs
Data Prepper ToC Update (#2514) * Creating PR with first file. Signed-off-by: carolxob <carolxob@amazon.com> * Adding newly created files to PR. Signed-off-by: carolxob <carolxob@amazon.com> * Reorganized files and added appropriate metadata to map ToC correctly. Signed-off-by: carolxob <carolxob@amazon.com> * Moved Authoring pipelines page. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates. Signed-off-by: carolxob <carolxob@amazon.com> * Minor ToC updates to Sources section for Data Prepper. Signed-off-by: carolxob <carolxob@amazon.com> * Updated Buffers section under Data Prepper. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update to otelmetricssource. Signed-off-by: carolxob <carolxob@amazon.com> * Restructured ToC in Processors section for Data Prepper. Signed-off-by: carolxob <carolxob@amazon.com> * Minor filename change. Signed-off-by: carolxob <carolxob@amazon.com> * Adjustments to metadata in ToC. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edit. Signed-off-by: carolxob <carolxob@amazon.com> * Fixed nav order in metadata. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edit. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update top metadata for ToC. Signed-off-by: carolxob <carolxob@amazon.com> * Adjustmenets to Toc order. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustments to ToC metadata. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustments to Sinks section. Signed-off-by: carolxob <carolxob@amazon.com> * Adjustements to high level ToC. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustement to Pipelines.md Signed-off-by: carolxob <carolxob@amazon.com> * Minor update. Signed-off-by: carolxob <carolxob@amazon.com> * Slight reorganization. Removed two placeholder pages for now. Signed-off-by: carolxob <carolxob@amazon.com> * Removed a page and replaced with pipelines content. Signed-off-by: carolxob <carolxob@amazon.com> * Minor changes/additions to content for placeholder pages. Signed-off-by: carolxob <carolxob@amazon.com> * Minor update to page link. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustments to ToC metadata. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Removed /clients from redirects to correct nav order. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustments to ToC metadata. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustments. Signed-off-by: carolxob <carolxob@amazon.com> * Minor adjustment ot metadata. Signed-off-by: carolxob <carolxob@amazon.com> * TOC link fixes Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Changed page name. Signed-off-by: carolxob <carolxob@amazon.com> * Corrected references to Peer Forwarder. Signed-off-by: carolxob <carolxob@amazon.com> * Renamed Data Prepper folder. Signed-off-by: carolxob <carolxob@amazon.com> * Minor updates to phrasing and capitalization. Signed-off-by: carolxob <carolxob@amazon.com> * Minor phrasing update. Signed-off-by: carolxob <carolxob@amazon.com> * Minor phrasing update. Signed-off-by: carolxob <carolxob@amazon.com> * Minor change. Signed-off-by: carolxob <carolxob@amazon.com> * Minor change to change S3 Source to S3Source. Signed-off-by: carolxob <carolxob@amazon.com> * Updated references to peer forwarder and changed capitalization. Signed-off-by: carolxob <carolxob@amazon.com> * Updated capitalization for peer forwarder. Signed-off-by: carolxob <carolxob@amazon.com> * Made edits based on doc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Update to one word. Signed-off-by: carolxob <carolxob@amazon.com> --------- Signed-off-by: carolxob <carolxob@amazon.com> Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Co-authored-by: Naarcha-AWS <naarcha@amazon.com>
2023-02-03 17:06:10 -05:00
parent: Managing Data Prepper
nav_order: 15
Add Core APIs to doc website repo. (#1932) * Adding Core API doc to doc website repo. Signed-off-by: carolxob <carolxob@amazon.com> * Made edits based on doc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Update _data-prepper/core-apis.md Co-authored-by: Alice Williams <88908598+alicejw-aws@users.noreply.github.com> * Minor edits based on doc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Pushing first set of updates because last ones did not commit. Signed-off-by: carolxob <carolxob@amazon.com> * Made more changes as part of editorial review. Signed-off-by: carolxob <carolxob@amazon.com> * Added more editorial feedback edits. Signed-off-by: carolxob <carolxob@amazon.com> * Pushing more editorial feedback changes. Signed-off-by: carolxob <carolxob@amazon.com> * Added more editorial feedback changes. Signed-off-by: carolxob <carolxob@amazon.com> * Added more editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Made slight changes. Signed-off-by: carolxob <carolxob@amazon.com> * Picked up a few more changes from editorial. Signed-off-by: carolxob <carolxob@amazon.com> * Incorporated doc review feedback. Needs a couple of tech questions addressed. Signed-off-by: carolxob <carolxob@amazon.com> * Minor changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits made from editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor capitalization update. Signed-off-by: carolxob <carolxob@amazon.com> * Update _data-prepper/core-apis.md * Update _data-prepper/core-apis.md Co-authored-by: Nate Bower <nbower@amazon.com> * Made changes based on editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> Signed-off-by: carolxob <carolxob@amazon.com> Co-authored-by: Alice Williams <88908598+alicejw-aws@users.noreply.github.com> Co-authored-by: Nate Bower <nbower@amazon.com>
2023-01-13 13:52:24 -05:00
---
# Core APIs
All Data Prepper instances expose a server with some control APIs. By default, this server runs on port 4900. Some plugins, especially source plugins, may expose other servers that run on different ports. Configurations for these plugins are independent of the core API. For example, to shut down Data Prepper, you can run the following curl request:
```
curl -X POST http://localhost:4900/shutdown
```
## APIs
The following table lists the available APIs.
| Name | Description |
| --- | --- |
| ```GET /list```<br>```POST /list``` | Returns a list of running pipelines. |
| ```POST /shutdown``` | Starts a graceful shutdown of Data Prepper. |
| ```GET /metrics/prometheus```<br>```POST /metrics/prometheus``` | Returns a scrape of Data Prepper metrics in Prometheus text format. This API is available as a `metricsRegistries` parameter in the Data Prepper configuration file `data-prepper-config.yaml` and contains `Prometheus` as part of the registry.
| ```GET /metrics/sys```<br>```POST /metrics/sys``` | Returns JVM metrics in Prometheus text format. This API is available as a `metricsRegistries` parameter in the Data Prepper configuration file `data-prepper-config.yaml` and contains `Prometheus` as part of the registry.
## Configuring the server
You can configure your Data Prepper core APIs through the `data-prepper-config.yaml` file.
### SSL/TLS connection
Many of the getting started guides for this project disable SSL on the endpoint:
```yaml
ssl: false
```
To enable SSL on your Data Prepper endpoint, configure your `data-prepper-config.yaml` file with the following options:
```yaml
ssl: true
keyStoreFilePath: "/usr/share/data-prepper/keystore.p12"
keyStorePassword: "secret"
privateKeyPassword: "secret"
```
For more information about configuring your Data Prepper server with SSL, see [Server Configuration](https://github.com/opensearch-project/data-prepper/blob/main/docs/configuration.md#server-configuration). If you are using a self-signed certificate, you can add the `-k` flag to the request to quickly test core APIs with SSL. Use the following `shutdown` request to test core APIs with SSL:
```
curl -k -X POST https://localhost:4900/shutdown
```
### Authentication
The Data Prepper core APIs support HTTP basic authentication. You can set the username and password with the following configuration in the `data-prepper-config.yaml` file:
```yaml
authentication:
http_basic:
username: "myuser"
password: "mys3cr3t"
```
You can disable authentication of core endpoints using the following configuration. Use this with caution because the shutdown API and others will be accessible to anybody with network access to your Data Prepper instance.
```yaml
authentication:
unauthenticated:
```
### Peer Forwarder
Peer Forwarder can be configured to enable stateful aggregation across multiple Data Prepper nodes. For more information about configuring Peer Forwarder, see [Peer forwarder]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/peer-forwarder/). It is supported by the `service_map_stateful`, `otel_trace_raw`, and `aggregate` processors.
Add Core APIs to doc website repo. (#1932) * Adding Core API doc to doc website repo. Signed-off-by: carolxob <carolxob@amazon.com> * Made edits based on doc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Update _data-prepper/core-apis.md Co-authored-by: Alice Williams <88908598+alicejw-aws@users.noreply.github.com> * Minor edits based on doc review feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Pushing first set of updates because last ones did not commit. Signed-off-by: carolxob <carolxob@amazon.com> * Made more changes as part of editorial review. Signed-off-by: carolxob <carolxob@amazon.com> * Added more editorial feedback edits. Signed-off-by: carolxob <carolxob@amazon.com> * Pushing more editorial feedback changes. Signed-off-by: carolxob <carolxob@amazon.com> * Added more editorial feedback changes. Signed-off-by: carolxob <carolxob@amazon.com> * Added more editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Made slight changes. Signed-off-by: carolxob <carolxob@amazon.com> * Picked up a few more changes from editorial. Signed-off-by: carolxob <carolxob@amazon.com> * Incorporated doc review feedback. Needs a couple of tech questions addressed. Signed-off-by: carolxob <carolxob@amazon.com> * Minor changes. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits. Signed-off-by: carolxob <carolxob@amazon.com> * Minor edits made from editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> * Minor capitalization update. Signed-off-by: carolxob <carolxob@amazon.com> * Update _data-prepper/core-apis.md * Update _data-prepper/core-apis.md Co-authored-by: Nate Bower <nbower@amazon.com> * Made changes based on editorial feedback. Signed-off-by: carolxob <carolxob@amazon.com> Signed-off-by: carolxob <carolxob@amazon.com> Co-authored-by: Alice Williams <88908598+alicejw-aws@users.noreply.github.com> Co-authored-by: Nate Bower <nbower@amazon.com>
2023-01-13 13:52:24 -05:00
### Shutdown timeouts
When you run the Data Prepper `shutdown` API, the process gracefully shuts down and clears any remaining data for both the `ExecutorService` sink and `ExecutorService` processor. The default timeout for shutdown of both processes is 10 seconds. You can configure the timeout with the following optional `data-prepper-config.yaml` file parameters:
```yaml
processorShutdownTimeout: "PT15M"
sinkShutdownTimeout: 30s
```
The values for these parameters are parsed into a `Duration` object through the [Data Prepper Duration Deserializer](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-core/src/main/java/org/opensearch/dataprepper/parser/DataPrepperDurationDeserializer.java).