MAINT: Data Prepper opensearch source metrics (#6392)

* MAINT: opensearch source metrics

Signed-off-by: George Chen <qchea@amazon.com>

* MNT: rewording

Signed-off-by: George Chen <qchea@amazon.com>

* MAINT: correct typo and rewording

Signed-off-by: George Chen <qchea@amazon.com>

* Update _data-prepper/pipelines/configuration/sources/opensearch.md

Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Qi Chen <chenqi0805@gmail.com>

* Update _data-prepper/pipelines/configuration/sources/opensearch.md

Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Qi Chen <chenqi0805@gmail.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

---------

Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Qi Chen <chenqi0805@gmail.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
This commit is contained in:
Qi Chen 2024-02-16 12:08:49 -06:00 committed by GitHub
parent eacc70b361
commit 74e50ca9fc
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 20 additions and 0 deletions

View File

@ -200,6 +200,26 @@ Option | Required | Type | Description
`sts_role_arn` | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to Amazon OpenSearch Service and Amazon OpenSearch Serverless. Default is `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html).
`serverless` | No | Boolean | Should be set to `true` when processing from an Amazon OpenSearch Serverless collection. Defaults to `false`.
## Metrics
The `opensearch` source includes the following metrics.
### Counters
- `documentsProcessed`: Measures the total number of documents processed by the `opensearch` source plugin.
- `indicesProcessed`: Measures the total number of indexes processed by the `opensearch` source plugin.
- `processingErrors`: Measures the total number of index processing errors made by the `opensearch` source plugin.
- `credentialsChanged`: Measures the number of times that the `opensearch` source refreshes basic credentials (username/password).
- `clientRefreshErrors`: Measures the number of errors encountered when generating a new client due to the `opensearch` source refreshing basic credentials.
### Timers
- `indexProcessingTime`: Measures the `opensearch` source plugin index processing latency, in seconds.
### Distribution summaries
- `bytesReceived`: Measures the size distribution of incoming documents, in bytes, received by the `opensearch` source plugin.
- `bytesProcessed`: Measures the size distribution of incoming document, in bytes, successfully processed by the `opensearch` source plugin.
## OpenSearch cluster security