From 74e50ca9fc184360a54c86afc6de26f3f2754d0f Mon Sep 17 00:00:00 2001 From: Qi Chen Date: Fri, 16 Feb 2024 12:08:49 -0600 Subject: [PATCH] MAINT: Data Prepper opensearch source metrics (#6392) * MAINT: opensearch source metrics Signed-off-by: George Chen * MNT: rewording Signed-off-by: George Chen * MAINT: correct typo and rewording Signed-off-by: George Chen * Update _data-prepper/pipelines/configuration/sources/opensearch.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Qi Chen * Update _data-prepper/pipelines/configuration/sources/opensearch.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Qi Chen * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: George Chen Signed-off-by: Qi Chen Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower --- .../configuration/sources/opensearch.md | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/_data-prepper/pipelines/configuration/sources/opensearch.md b/_data-prepper/pipelines/configuration/sources/opensearch.md index b4b3ddc6..7cc0b9a3 100644 --- a/_data-prepper/pipelines/configuration/sources/opensearch.md +++ b/_data-prepper/pipelines/configuration/sources/opensearch.md @@ -200,6 +200,26 @@ Option | Required | Type | Description `sts_role_arn` | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to Amazon OpenSearch Service and Amazon OpenSearch Serverless. Default is `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). `serverless` | No | Boolean | Should be set to `true` when processing from an Amazon OpenSearch Serverless collection. Defaults to `false`. +## Metrics + +The `opensearch` source includes the following metrics. + +### Counters + +- `documentsProcessed`: Measures the total number of documents processed by the `opensearch` source plugin. +- `indicesProcessed`: Measures the total number of indexes processed by the `opensearch` source plugin. +- `processingErrors`: Measures the total number of index processing errors made by the `opensearch` source plugin. +- `credentialsChanged`: Measures the number of times that the `opensearch` source refreshes basic credentials (username/password). +- `clientRefreshErrors`: Measures the number of errors encountered when generating a new client due to the `opensearch` source refreshing basic credentials. + +### Timers + +- `indexProcessingTime`: Measures the `opensearch` source plugin index processing latency, in seconds. + +### Distribution summaries + +- `bytesReceived`: Measures the size distribution of incoming documents, in bytes, received by the `opensearch` source plugin. +- `bytesProcessed`: Measures the size distribution of incoming document, in bytes, successfully processed by the `opensearch` source plugin. ## OpenSearch cluster security