diff --git a/_clients/agents-and-ingestion-tools/index.md b/_clients/agents-and-ingestion-tools/index.md index 5e3edad1..e131e805 100644 --- a/_clients/agents-and-ingestion-tools/index.md +++ b/_clients/agents-and-ingestion-tools/index.md @@ -1,7 +1,7 @@ --- layout: default title: Agents and ingestion tools -nav_order: 100 +nav_order: 140 has_children: false has_toc: false redirect_from: diff --git a/_observability/data-prepper/data-prepper-reference.md b/_clients/data-prepper/data-prepper-reference.md similarity index 98% rename from _observability/data-prepper/data-prepper-reference.md rename to _clients/data-prepper/data-prepper-reference.md index f884bd1a..3a8d023b 100644 --- a/_observability/data-prepper/data-prepper-reference.md +++ b/_clients/data-prepper/data-prepper-reference.md @@ -7,7 +7,7 @@ nav_order: 3 # Data Prepper configuration reference -This page lists all supported Data Prepper server, sources, buffers, preppers, and sinks, along with their associated options. For example configuration files, see [Data Prepper]({{site.url}}{{site.baseurl}}/observability/data-prepper/pipelines/). +This page lists all supported Data Prepper server, sources, buffers, preppers, and sinks, along with their associated options. For example configuration files, see [Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/pipelines/). ## Data Prepper server options @@ -49,7 +49,7 @@ max_connection_count | No | Integer | The maximum allowed number of open connect ssl | No | Boolea | Enables connections to the OTel source port over TLS/SSL. Defaults to `true`. sslKeyCertChainFile | Conditionally | String | File-system path or AWS S3 path to the security certificate (e.g. `"config/demo-data-prepper.crt"` or `"s3://my-secrets-bucket/demo-data-prepper.crt"`). Required if ssl is set to `true`. sslKeyFile | Conditionally | String | File-system path or AWS S3 path to the security key (e.g. `"config/demo-data-prepper.key"` or `"s3://my-secrets-bucket/demo-data-prepper.key"`). Required if ssl is set to `true`. -useAcmCertForSSL | No | Boolean, enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. +useAcmCertForSSL | No | Boolean | Whether to enable TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. acmCertificateArn | Conditionally | String | Represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `useAcmCertForSSL` is set to `true`. awsRegion | Conditionally | String | Represents the AWS region to use ACM or S3. Required if `useAcmCertForSSL` is set to `true` or `sslKeyCertChainFile` and `sslKeyFile` are AWS S3 paths. authentication | No | Object| An authentication configuration. By default, this runs an unauthenticated server. This uses pluggable authentication for HTTPS. To use basic authentication, define the `http_basic` plugin with a `username` and `password`. To provide customer authentication use or create a plugin which implements: [GrpcAuthenticationProvider](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/armeria-common/src/main/java/com/amazon/dataprepper/armeria/authentication/GrpcAuthenticationProvider.java). diff --git a/_observability/data-prepper/get-started.md b/_clients/data-prepper/get-started.md similarity index 96% rename from _observability/data-prepper/get-started.md rename to _clients/data-prepper/get-started.md index 4b6a1363..11ef4ea9 100644 --- a/_observability/data-prepper/get-started.md +++ b/_clients/data-prepper/get-started.md @@ -41,7 +41,7 @@ docker run --name data-prepper \ opensearchproject/opensearch-data-prepper:latest ``` -This sample pipeline configuration above demonstrates a simple pipeline with a source (`random`) sending data to a sink (`stdout`). For more examples and details on more advanced pipeline configurations, see [Pipelines]({{site.url}}{{site.baseurl}}/observability/data-prepper/pipelines). +This sample pipeline configuration above demonstrates a simple pipeline with a source (`random`) sending data to a sink (`stdout`). For more examples and details on more advanced pipeline configurations, see [Pipelines]({{site.url}}{{site.baseurl}}/clients/data-prepper/pipelines). After starting Data Prepper, you should see log output and some UUIDs after a few seconds: diff --git a/_observability/data-prepper/index.md b/_clients/data-prepper/index.md similarity index 56% rename from _observability/data-prepper/index.md rename to _clients/data-prepper/index.md index f2d0644e..7fb833f4 100644 --- a/_observability/data-prepper/index.md +++ b/_clients/data-prepper/index.md @@ -1,7 +1,7 @@ --- layout: default title: Data Prepper -nav_order: 80 +nav_order: 120 has_children: true has_toc: false --- @@ -10,6 +10,6 @@ has_toc: false Data Prepper is a server side data collector capable of filtering, enriching, transforming, normalizing and aggregating data for downstream analytics and visualization. -Data Prepper lets users build custom pipelines to improve the operational view of applications. Two common uses for Data Prepper are trace and log analytics. [Trace analytics]({{site.url}}{{site.baseurl}}/observability/trace/index/) can help you visualize the flow of events and identify performance problems, and [log analytics]({{site.url}}{{site.baseurl}}/observability/log-analytics/) can improve searching, analyzing and provide insights into your application. +Data Prepper lets users build custom pipelines to improve the operational view of applications. Two common uses for Data Prepper are trace and log analytics. [Trace analytics]({{site.url}}{{site.baseurl}}/observability-plugin/trace/index/) can help you visualize the flow of events and identify performance problems, and [log analytics]({{site.url}}{{site.baseurl}}/observability-plugin/log-analytics/) can improve searching, analyzing and provide insights into your application. -To get started building your own custom pipelines with Data Prepper, see the [Get Started]({{site.url}}{{site.baseurl}}/observability/data-prepper/get-started/) guide. +To get started building your own custom pipelines with Data Prepper, see the [Get Started]({{site.url}}{{site.baseurl}}/clients/data-prepper/get-started/) guide. diff --git a/_observability/data-prepper/pipelines.md b/_clients/data-prepper/pipelines.md similarity index 94% rename from _observability/data-prepper/pipelines.md rename to _clients/data-prepper/pipelines.md index 32f5e59a..b664d98a 100644 --- a/_observability/data-prepper/pipelines.md +++ b/_clients/data-prepper/pipelines.md @@ -42,7 +42,7 @@ simple-sample-pipeline: ## Examples -This section provides some pipeline examples that you can use to start creating your own pipelines. For more information, see [Data Prepper configuration reference]({{site.url}}{{site.baseurl}}/observability/data-prepper/data-prepper-reference/) guide. +This section provides some pipeline examples that you can use to start creating your own pipelines. For more information, see [Data Prepper configuration reference]({{site.url}}{{site.baseurl}}/clients/data-prepper/data-prepper-reference/) guide. The Data Prepper repository has several [sample applications](https://github.com/opensearch-project/data-prepper/tree/main/examples) to help you get started. @@ -73,7 +73,7 @@ This example uses weak security. We strongly recommend securing all plugins whic ### Trace Analytics pipeline -The following example demonstrates how to build a pipeline that supports the [Trace Analytics OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observability/trace/ta-dashboards/). This pipeline takes data from the OpenTelemetry Collector and uses two other pipelines as sinks. These two separate pipelines index trace and the service map documents for the dashboard plugin. +The following example demonstrates how to build a pipeline that supports the [Trace Analytics OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observability-plugin/trace/ta-dashboards/). This pipeline takes data from the OpenTelemetry Collector and uses two other pipelines as sinks. These two separate pipelines index trace and the service map documents for the dashboard plugin. ```yml entry-pipeline: diff --git a/_clients/javascript.md b/_clients/javascript.md index c670e4b8..45923e05 100644 --- a/_clients/javascript.md +++ b/_clients/javascript.md @@ -1,7 +1,7 @@ --- layout: default title: JavaScript client -nav_order: 90 +nav_order: 100 --- # JavaScript client diff --git a/_config.yml b/_config.yml index df098fc2..2d609a75 100644 --- a/_config.yml +++ b/_config.yml @@ -49,7 +49,7 @@ collections: replication-plugin: permalink: /:collection/:path/ output: true - observability: + observability-plugin: permalink: /:collection/:path/ output: true monitoring-plugins: @@ -91,8 +91,8 @@ just_the_docs: replication-plugin: name: Replication plugin nav_fold: true - observability: - name: Observability + observability-plugin: + name: Observability plugin nav_fold: true monitoring-plugins: name: Monitoring plugins diff --git a/_observability/event-analytics.md b/_observability-plugin/event-analytics.md similarity index 84% rename from _observability/event-analytics.md rename to _observability-plugin/event-analytics.md index 82b841fa..cdbe0ee5 100644 --- a/_observability/event-analytics.md +++ b/_observability-plugin/event-analytics.md @@ -6,7 +6,7 @@ nav_order: 10 # Event analytics -Event analytics in observability is where you can use [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability/ppl/index) (PPL) queries to build and view different visualizations of your data. +Event analytics in observability is where you can use [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) (PPL) queries to build and view different visualizations of your data. ## Get started with event analytics @@ -24,10 +24,10 @@ source = opensearch_dashboards_sample_data_logs | fields host | stats count() By default, Dashboards shows results from the last 15 minutes of your data. To see data from a different timeframe, use the date and time selector. -For more information about building PPL queries, see [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability/ppl/index). +For more information about building PPL queries, see [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index). ## Save a visualization -After Dashboards generates a visualization, you must save it if you want to return to it at a later time or if you want to add it to an [operational panel]({{site.url}}{{site.baseurl}}/observability/operational-panels). +After Dashboards generates a visualization, you must save it if you want to return to it at a later time or if you want to add it to an [operational panel]({{site.url}}{{site.baseurl}}/observability-plugin/operational-panels). To save a visualization, expand the save dropdown menu next to **Run**, enter a name for your visualization, then choose **Save**. You can reopen any saved visualizations on the event analytics page. diff --git a/_observability/index.md b/_observability-plugin/index.md similarity index 70% rename from _observability/index.md rename to _observability-plugin/index.md index 918ad21b..b7d05fe9 100644 --- a/_observability/index.md +++ b/_observability-plugin/index.md @@ -4,8 +4,8 @@ title: About Observability nav_order: 1 has_children: false redirect_from: - - /observability/ - - /observability/ + - /observability-plugin/ + - /observability-plugin/ --- # About Observability @@ -16,13 +16,13 @@ Observability is collection of plugins and applications that let you visualize d Your experience of exploring data might differ, but if you're new to exploring data to create visualizations, we recommend trying a workflow like the following: -1. Explore data over a certain timeframe using [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability/ppl/index). -2. Use [event analytics]({{site.url}}{{site.baseurl}}/observability/event-analytics) to turn data-driven events into visualizations. +1. Explore data over a certain timeframe using [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index). +2. Use [event analytics]({{site.url}}{{site.baseurl}}/observability-plugin/event-analytics) to turn data-driven events into visualizations. ![Sample Event Analytics View]({{site.url}}{{site.baseurl}}/images/event-analytics.png) -3. Create [operational panels]({{site.url}}{{site.baseurl}}/observability/operational-panels) and add visualizations to compare data the way you like. +3. Create [operational panels]({{site.url}}{{site.baseurl}}/observability-plugin/operational-panels) and add visualizations to compare data the way you like. ![Sample Operational Panel View]({{site.url}}{{site.baseurl}}/images/operational-panel.png) -4. Use [log analytics]({{site.url}}{{site.baseurl}}/observability/log-analytics) to transform unstructured log data. -5. Use [trace analytics]({{site.url}}{{site.baseurl}}/observability/trace/index) to create traces and dive deep into your data. +4. Use [log analytics]({{site.url}}{{site.baseurl}}/observability-plugin/log-analytics) to transform unstructured log data. +5. Use [trace analytics]({{site.url}}{{site.baseurl}}/observability-plugin/trace/index) to create traces and dive deep into your data. ![Sample Trace Analytics View]({{site.url}}{{site.baseurl}}/images/observability-trace.png) -6. Leverage [notebooks]({{site.url}}{{site.baseurl}}/observability/notebooks) to combine different visualizations and code blocks that you can share with team members. +6. Leverage [notebooks]({{site.url}}{{site.baseurl}}/observability-plugin/notebooks) to combine different visualizations and code blocks that you can share with team members. ![Sample Notebooks View]({{site.url}}{{site.baseurl}}/images/notebooks.png) diff --git a/_observability/log-analytics.md b/_observability-plugin/log-analytics.md similarity index 86% rename from _observability/log-analytics.md rename to _observability-plugin/log-analytics.md index b7a9c2dd..8128d6d5 100644 --- a/_observability/log-analytics.md +++ b/_observability-plugin/log-analytics.md @@ -10,7 +10,7 @@ Log ingestion provides a way to transform unstructured log data into structured ## Get started with log ingestion -OpenSearch Log Ingestion consists of three components---[Data Prepper]({{site.url}}{{site.baseurl}}/observability/data-prepper/index/), [OpenSearch]({{site.url}}{{site.baseurl}}/) and [OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/)---that fit into the OpenSearch ecosystem. The Data Prepper repository has several [sample applications](https://github.com/opensearch-project/data-prepper/tree/main/examples) to help you get started. +OpenSearch Log Ingestion consists of three components---[Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/index/), [OpenSearch]({{site.url}}{{site.baseurl}}/) and [OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/)---that fit into the OpenSearch ecosystem. The Data Prepper repository has several [sample applications](https://github.com/opensearch-project/data-prepper/tree/main/examples) to help you get started. ### Basic flow of data @@ -20,7 +20,7 @@ OpenSearch Log Ingestion consists of three components---[Data Prepper]({{site.ur (In the [example](#example) below, [FluentBit](https://docs.fluentbit.io/manual/) is used as a log collector that collects log data from a file and sends the log data to Data Prepper). -2. [Data Prepper]({{site.url}}{{site.baseurl}}/observability/data-prepper/index/) receives the log data, transforms the data into a structure format, and indexes it on an OpenSearch cluster. +2. [Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/index/) receives the log data, transforms the data into a structure format, and indexes it on an OpenSearch cluster. 3. The data can then be explored through OpenSearch search queries or the **Discover** page in OpenSearch Dashboards. diff --git a/_observability/notebooks.md b/_observability-plugin/notebooks.md similarity index 100% rename from _observability/notebooks.md rename to _observability-plugin/notebooks.md diff --git a/_observability/operational-panels.md b/_observability-plugin/operational-panels.md similarity index 86% rename from _observability/operational-panels.md rename to _observability-plugin/operational-panels.md index b037dfe4..9578d7cd 100644 --- a/_observability/operational-panels.md +++ b/_observability-plugin/operational-panels.md @@ -6,7 +6,7 @@ nav_order: 30 # Operational panels -Operational panels in OpenSearch Dashboards are collections of visualizations generated using [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability/ppl/index) (PPL) queries. +Operational panels in OpenSearch Dashboards are collections of visualizations generated using [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) (PPL) queries. ## Get started with operational panels @@ -16,7 +16,7 @@ If you want to start using operational panels without adding any data, expand th To create an operational panel and add visualizations: -1. From the **Add Visualization** dropdown menu, choose **Select Existing Visualization** or **Create New Visualization**, which takes you to the [event analytics]({{site.url}}{{site.baseurl}}/observability/event-analytics) explorer, where you can use PPL to create visualizations. +1. From the **Add Visualization** dropdown menu, choose **Select Existing Visualization** or **Create New Visualization**, which takes you to the [event analytics]({{site.url}}{{site.baseurl}}/observability-plugin/event-analytics) explorer, where you can use PPL to create visualizations. 1. If you're adding already existing visualizations, choose a visualization from the dropdown menu. 1. Choose **Add**. diff --git a/_observability/ppl/commands.md b/_observability-plugin/ppl/commands.md similarity index 100% rename from _observability/ppl/commands.md rename to _observability-plugin/ppl/commands.md diff --git a/_observability/ppl/datatypes.md b/_observability-plugin/ppl/datatypes.md similarity index 100% rename from _observability/ppl/datatypes.md rename to _observability-plugin/ppl/datatypes.md diff --git a/_observability/ppl/endpoint.md b/_observability-plugin/ppl/endpoint.md similarity index 100% rename from _observability/ppl/endpoint.md rename to _observability-plugin/ppl/endpoint.md diff --git a/_observability/ppl/functions.md b/_observability-plugin/ppl/functions.md similarity index 100% rename from _observability/ppl/functions.md rename to _observability-plugin/ppl/functions.md diff --git a/_observability/ppl/identifiers.md b/_observability-plugin/ppl/identifiers.md similarity index 100% rename from _observability/ppl/identifiers.md rename to _observability-plugin/ppl/identifiers.md diff --git a/_observability/ppl/index.md b/_observability-plugin/ppl/index.md similarity index 100% rename from _observability/ppl/index.md rename to _observability-plugin/ppl/index.md diff --git a/_observability/ppl/protocol.md b/_observability-plugin/ppl/protocol.md similarity index 100% rename from _observability/ppl/protocol.md rename to _observability-plugin/ppl/protocol.md diff --git a/_observability/ppl/settings.md b/_observability-plugin/ppl/settings.md similarity index 100% rename from _observability/ppl/settings.md rename to _observability-plugin/ppl/settings.md diff --git a/_observability/trace/get-started.md b/_observability-plugin/trace/get-started.md similarity index 93% rename from _observability/trace/get-started.md rename to _observability-plugin/trace/get-started.md index a9808d88..852626a9 100644 --- a/_observability/trace/get-started.md +++ b/_observability-plugin/trace/get-started.md @@ -19,9 +19,9 @@ OpenSearch Trace Analytics consists of two components---Data Prepper and the Tra 1. The [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/getting-started/) receives data from the application and formats it into OpenTelemetry data. -1. [Data Prepper]({{site.url}}{{site.baseurl}}/observability/data-prepper/index/) processes the OpenTelemetry data, transforms it for use in OpenSearch, and indexes it on an OpenSearch cluster. +1. [Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/index/) processes the OpenTelemetry data, transforms it for use in OpenSearch, and indexes it on an OpenSearch cluster. -1. The [Trace Analytics OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observability/trace/ta-dashboards/) displays the data in near real-time as a series of charts and tables, with an emphasis on service architecture, latency, error rate, and throughput. +1. The [Trace Analytics OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observability-plugin/trace/ta-dashboards/) displays the data in near real-time as a series of charts and tables, with an emphasis on service architecture, latency, error rate, and throughput. ## Jaeger HotROD @@ -78,4 +78,4 @@ curl -X GET -u 'admin:admin' -k 'https://localhost:9200/otel-v1-apm-span-000001/ Navigate to `http://localhost:5601` in a web browser and choose **Trace Analytics**. You can see the results of your single click in the Jaeger HotROD web interface: the number of traces per API and HTTP method, latency trends, a color-coded map of the service architecture, and a list of trace IDs that you can use to drill down on individual operations. -If you don't see your trace, adjust the timeframe in OpenSearch Dashboards. For more information on using the plugin, see [OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observability/trace/ta-dashboards/). +If you don't see your trace, adjust the timeframe in OpenSearch Dashboards. For more information on using the plugin, see [OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observability-plugin/trace/ta-dashboards/). diff --git a/_observability/trace/index.md b/_observability-plugin/trace/index.md similarity index 100% rename from _observability/trace/index.md rename to _observability-plugin/trace/index.md diff --git a/_observability/trace/ta-dashboards.md b/_observability-plugin/trace/ta-dashboards.md similarity index 100% rename from _observability/trace/ta-dashboards.md rename to _observability-plugin/trace/ta-dashboards.md diff --git a/_opensearch/data-streams.md b/_opensearch/data-streams.md index 7fa5fcc0..69ee1b13 100644 --- a/_opensearch/data-streams.md +++ b/_opensearch/data-streams.md @@ -262,4 +262,4 @@ You can use wildcards to delete more than one data stream. We recommend deleting data from a data stream using an ISM policy. -You can also use [asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/index/) and [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/) and [PPL]({{site.url}}{{site.baseurl}}/observability/ppl/index/) to query your data stream directly. You can also use the security plugin to define granular permissions on the data stream name. +You can also use [asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/index/) and [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/) and [PPL]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index/) to query your data stream directly. You can also use the security plugin to define granular permissions on the data stream name.