From 78a72f5d22a9f7594409c40d4f7b03d03390b2e6 Mon Sep 17 00:00:00 2001 From: Eli Fisher Date: Fri, 4 Jun 2021 15:34:24 -0400 Subject: [PATCH 1/5] Adds Agensts and Ingestion Tools page with compatibility matrices --- docs/agents-and-ingestion-tools/index.md | 64 ++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 docs/agents-and-ingestion-tools/index.md diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md new file mode 100644 index 00000000..51b04902 --- /dev/null +++ b/docs/agents-and-ingestion-tools/index.md @@ -0,0 +1,64 @@ +--- +layout: default +title: Agents and Ingestion Tools +nav_order: 100 +has_children: false +has_toc: false +--- + +# Agents and Ingestion Tools + +Historically there have been multiple popular agents and ingestion tools that work with Elasticsearch. Some of which are Beats, Logstash, Fluentd, FluentBit, and Open Telemetry. OpenSearch aims to continue to support a broad set of agents and ingestion tools, but not all have been tested or have explicitly added OpenSearch compatibility. To make getting started with OpenSearch easier a [set version YML configuration](https://github.com/opensearch-project/OpenSearch/issues/693) is being added so that you can set OpenSearch 1.x clusters to report open source Elasticsearch 7.10.2. This will enable it to connect with tools that do version checking while need OpenSource compatibilities are added to your tools. + + +## Compatibility Matrices + +So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page okease [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) that includes them and feel free to add yourself to the Credits and Thanks you at the bottom of the page. We greatly appreciate the help!** + +Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [planned OpenSearch output plugin](). + +### Compatibility Matrix for Logstash + +| |Logstash OSS 7.x to 7.11.x* |Logstash OSS 7.12.x** |Logstash 7.13.x without OpenSearch output plugin^ |Logstash 7.13.x with OpenSearch output plugin***^ | +|--- |--- |--- |--- |--- | +|Elasticsearch OSS v7.x to v7.9.x |*Yes* |*Yes* |*No* |*Yes* | +|Elasticsearch OSS v7.10.2^ |*Yes* |*Yes* |*No* |*Yes* | +|ODFE OSS v1.x to 1.12 |*Yes* |*Yes* |*No* |*Yes* | +|ODFE 1.13^ |*Yes* |*Yes* |*No* |*Yes* | +|Amazon Elasticsearch Service 7.x to 7.9 |Yes |Yes |*No* |*Yes* | +|Amazon Elasticsearch Service 7.10^ |Yes |Yes |*No* |*Yes* | +|Amazon Elasticsearch Service 7.x to 7.9 with IAM |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |No |*Yes* | +|Amazon Elasticsearch Service 7.10 with IAM^ |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |No |*Yes* | +|OpenSearch 1.0^ |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |*No* |*Yes* | + +*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat) +**Most current compatible version with Elasticsearch OSS +***Planning to build +^ Most current version of software +*Italicized cells are based on documentation or release notes but are still pending testing to validate* + +### Compatibility Matrix for Beats + +| |Beats OSS 7.x to 7.11.x* |Beats OSS 7.12.x** |Beats 7.13.x^ | +|--- |--- |--- |--- | +|Elasticsearch OSS v7.x to v7.9.x |*Yes* |*Yes* |No | +|Elasticsearch OSS v7.10.2^ |*Yes* |*Yes* |No | +|ODFE OSS v1.x to 1.12 |*Yes* |*Yes* |No | +|ODFE 1.13^ |*Yes* |*Yes* |No | +|Amazon Elasticsearch Service 7.x to 7.9 |*Yes* |*Yes* |No | +|Amazon Elasticsearch Service 7.10^ |*Yes* |*Yes* |No | +|OpenSearch 1.0^ |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |No | +|Logstash OSS 7.x to 7.11.x* |*Yes* |*Yes* |*Yes* | +|Logstash OSS 7.12.x** |*Yes* |*Yes* |*Yes* | +|Logstash 7.13.x with OpenSearch output plugin |*Yes* |*Yes* |*Yes* | + +*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat) +**Most current compatible version with Elasticsearch OSS +^ Most current version of software +*Italicized cells are based on documentation or release notes but are still pending testing to validate* + +### Credits and Thank You for Contributing and Testing + +* [VijayanB](https://github.com/VijayanB) +* [vamshin](https://github.com/vamshin) +* [anirudha](https://github.com/anirudha) From 0775042a7ded893a67a774fc16d2c0afe3f7e514 Mon Sep 17 00:00:00 2001 From: Eli Fisher Date: Fri, 4 Jun 2021 15:56:31 -0400 Subject: [PATCH 2/5] fixes missing link to OpenSearch output plugin issue --- docs/agents-and-ingestion-tools/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md index 51b04902..ae277514 100644 --- a/docs/agents-and-ingestion-tools/index.md +++ b/docs/agents-and-ingestion-tools/index.md @@ -15,7 +15,7 @@ Historically there have been multiple popular agents and ingestion tools that wo So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page okease [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) that includes them and feel free to add yourself to the Credits and Thanks you at the bottom of the page. We greatly appreciate the help!** -Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [planned OpenSearch output plugin](). +Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [planned OpenSearch output plugin](https://github.com/opensearch-project/OpenSearch/issues/820). ### Compatibility Matrix for Logstash From 7190744ae79aa9d2fd82c12f74662199d3e921a5 Mon Sep 17 00:00:00 2001 From: Eli Fisher Date: Fri, 4 Jun 2021 16:01:23 -0400 Subject: [PATCH 3/5] Fixes formatting of content below tables --- docs/agents-and-ingestion-tools/index.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md index ae277514..58218f40 100644 --- a/docs/agents-and-ingestion-tools/index.md +++ b/docs/agents-and-ingestion-tools/index.md @@ -31,10 +31,10 @@ Note that if a cell in any of the matrices is *italicized* that means the value |Amazon Elasticsearch Service 7.10 with IAM^ |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |No |*Yes* | |OpenSearch 1.0^ |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |*No* |*Yes* | -*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat) -**Most current compatible version with Elasticsearch OSS -***Planning to build -^ Most current version of software +\*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat)\ +\*\*Most current compatible version with Elasticsearch OSS\ +\*\*\*Planning to build\ +^ Most current version of software\ *Italicized cells are based on documentation or release notes but are still pending testing to validate* ### Compatibility Matrix for Beats @@ -52,9 +52,9 @@ Note that if a cell in any of the matrices is *italicized* that means the value |Logstash OSS 7.12.x** |*Yes* |*Yes* |*Yes* | |Logstash 7.13.x with OpenSearch output plugin |*Yes* |*Yes* |*Yes* | -*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat) -**Most current compatible version with Elasticsearch OSS -^ Most current version of software +\*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat)\ +\*\*Most current compatible version with Elasticsearch OSS\ +^ Most current version of software\ *Italicized cells are based on documentation or release notes but are still pending testing to validate* ### Credits and Thank You for Contributing and Testing From e066d01f6da7de4eca0790a3fdaacaae8bb7103b Mon Sep 17 00:00:00 2001 From: Eli Fisher Date: Fri, 4 Jun 2021 16:39:26 -0400 Subject: [PATCH 4/5] Fixes typo --- docs/agents-and-ingestion-tools/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md index 58218f40..b5d25c04 100644 --- a/docs/agents-and-ingestion-tools/index.md +++ b/docs/agents-and-ingestion-tools/index.md @@ -13,7 +13,7 @@ Historically there have been multiple popular agents and ingestion tools that wo ## Compatibility Matrices -So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page okease [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) that includes them and feel free to add yourself to the Credits and Thanks you at the bottom of the page. We greatly appreciate the help!** +So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page please [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) that includes them and feel free to add yourself to the Credits and Thanks you at the bottom of the page. We greatly appreciate the help!** Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [planned OpenSearch output plugin](https://github.com/opensearch-project/OpenSearch/issues/820). From be083bb5db2f9fa6a55860c94bf2bac7a7e7564a Mon Sep 17 00:00:00 2001 From: Eli Fisher Date: Fri, 4 Jun 2021 16:54:25 -0400 Subject: [PATCH 5/5] Fixes typos and tries to simplify some sentences --- docs/agents-and-ingestion-tools/index.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/docs/agents-and-ingestion-tools/index.md b/docs/agents-and-ingestion-tools/index.md index b5d25c04..cac4bb97 100644 --- a/docs/agents-and-ingestion-tools/index.md +++ b/docs/agents-and-ingestion-tools/index.md @@ -8,18 +8,18 @@ has_toc: false # Agents and Ingestion Tools -Historically there have been multiple popular agents and ingestion tools that work with Elasticsearch. Some of which are Beats, Logstash, Fluentd, FluentBit, and Open Telemetry. OpenSearch aims to continue to support a broad set of agents and ingestion tools, but not all have been tested or have explicitly added OpenSearch compatibility. To make getting started with OpenSearch easier a [set version YML configuration](https://github.com/opensearch-project/OpenSearch/issues/693) is being added so that you can set OpenSearch 1.x clusters to report open source Elasticsearch 7.10.2. This will enable it to connect with tools that do version checking while need OpenSource compatibilities are added to your tools. +Historically, there have been multiple popular agents and ingestion tools that work with Elasticsearch. Some of which are Beats, Logstash, Fluentd, FluentBit, and Open Telemetry. OpenSearch aims to continue to support a broad set of agents and ingestion tools, but not all have been tested or have explicitly added OpenSearch compatibility. To make getting started with OpenSearch easier, a [version value](https://github.com/opensearch-project/OpenSearch/issues/693) is being added to the OpenSearch YML. This will let you set OpenSearch 1.x clusters to report open source Elasticsearch 7.10.2. By reporting 7.10.2, the cluster will be able to connect with tools that do version checking. This is intended to be an intermediate solution while OpenSearch support is added to more tools. ## Compatibility Matrices -So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page please [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) that includes them and feel free to add yourself to the Credits and Thanks you at the bottom of the page. We greatly appreciate the help!** +So far we have built compatibility matrices for OpenSearch, Elasticsearch OSS 7.x, Open Distro for Elasticsearch (ODFE) 1.x, Logstash OSS 7.x, and Beats OSS 7.x. **This page is a living document. If there are other versions or software you would like to add to the page please [submit a PR](https://github.com/opensearch-project/documentation-website/edit/main/docs/agents-and-ingestion-tools/index.md) and add yourself to the "Credits and Thanks you" section at the bottom of the page. We greatly appreciate the help!** -Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [planned OpenSearch output plugin](https://github.com/opensearch-project/OpenSearch/issues/820). +Note that if a cell in any of the matrices is *italicized* that means the value is theoretically what it should be based on other documentation or release notes, but it is still pending testing. Also note that for Logstash we have included a column for version 7.13.x with the [proposed OpenSearch output plugin](https://github.com/opensearch-project/OpenSearch/issues/820). ### Compatibility Matrix for Logstash -| |Logstash OSS 7.x to 7.11.x* |Logstash OSS 7.12.x** |Logstash 7.13.x without OpenSearch output plugin^ |Logstash 7.13.x with OpenSearch output plugin***^ | +| |Logstash OSS 7.x to 7.11.x |Logstash OSS 7.12.x* |Logstash 7.13.x without OpenSearch output plugin^ |Logstash 7.13.x with OpenSearch output plugin**^ | |--- |--- |--- |--- |--- | |Elasticsearch OSS v7.x to v7.9.x |*Yes* |*Yes* |*No* |*Yes* | |Elasticsearch OSS v7.10.2^ |*Yes* |*Yes* |*No* |*Yes* | @@ -31,9 +31,8 @@ Note that if a cell in any of the matrices is *italicized* that means the value |Amazon Elasticsearch Service 7.10 with IAM^ |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |[Yes with Amazon ES output plugin](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-logstash.html) |No |*Yes* | |OpenSearch 1.0^ |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |[Yes via version setting](https://github.com/opensearch-project/OpenSearch/issues/693) |*No* |*Yes* | -\*Beats OSS includes all Apache 2.0 Beats agents (Filebeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, Packetbeat)\ -\*\*Most current compatible version with Elasticsearch OSS\ -\*\*\*Planning to build\ +\*Most current compatible version with Elasticsearch OSS\ +\*\*Planning to build\ ^ Most current version of software\ *Italicized cells are based on documentation or release notes but are still pending testing to validate*