minor changes

This commit is contained in:
ashwinkumar12345 2021-07-06 11:53:10 -07:00
parent 9648a850ad
commit 70c9b6d9c8
4 changed files with 15 additions and 14 deletions

View File

@ -11,7 +11,7 @@ This page contains a list of common filter plugins.
## mutate
You can use the `mutate` filter to change the data type of a field. For example, you can use the `mutate` filter if you're sending events to Opensearch and you need to change the data type of a field to match any existing mappings.
You can use the `mutate` filter to change the data type of a field. For example, you can use the `mutate` filter if you're sending events to OpenSearch and you need to change the data type of a field to match any existing mappings.
To convert the `quantity` field from a `string` type to an `integer` type:
@ -154,4 +154,4 @@ http_status: 200
num_bytes: 12798
```
For common log formats, you use the predefined patterns defined here - [Logstash patterns](https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/ecs-v1). You cam make any adjustments to the results with the `mutate` filter.
For common log formats, you use the predefined patterns defined here---[Logstash patterns](https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/ecs-v1). You can make any adjustments to the results with the `mutate` filter.

View File

@ -13,7 +13,7 @@ Here's a brief introduction to how Logstash processes events internally.
You can configure Logstash to have a number of inputs listening for events. Each input runs in its own thread to avoid inputs blocking each other. If you have two incoming events at the same time, Logstash handles both events concurrently.
After receiving an event and possibly applying an input codec, Logstash sends the event to a work queue. Pipeline workers or batchers perform the rest of work involving filters and outputs along with any codec used at the output. Each pipeline worker also runs within its own thread meaning that Logstash processes multiple events simultaneously.
After receiving an event and possibly applying an input codec, Logstash sends the event to a work queue. Pipeline workers or batchers perform the rest of the work involving filters and outputs along with any codec used at the output. Each pipeline worker also runs within its own thread meaning that Logstash processes multiple events simultaneously.
## Processing events in batches
@ -21,14 +21,14 @@ A pipeline worker consumes events from the work queue in batches to optimize the
One reason why Logstash works in batches is that some code needs to be executed regardless of how many events are processed at a time within the pipeline worker. Instead of executing that code 100 times for 100 events, its more efficient to execute it once for a batch of 100 events.
Another reason is that some output plugins group together events as batches. For example, if you send 100 requests to Opensearch, the Opensearch output plugin uses the bulk API to send a single request that groups together the 100 requests.
Another reason is that a few output plugins group together events as batches. For example, if you send 100 requests to OpenSearch, the OpenSearch output plugin uses the bulk API to send a single request that groups together the 100 requests.
Logstash determines the batch size by two configuration optionsa number representing the maximum batch size and the batch delay. The batch delay is how long Logstash waits before processing the unprocessed batch of events.
Logstash determines the batch size by two configuration options---a number representing the maximum batch size and the batch delay. The batch delay is how long Logstash waits before processing the unprocessed batch of events.
If you set the maximum batch size to 50 and the batch delay to 100 ms, Logstash processes a batch if they're either 50 unprocessed events in the work queue or if one hundred milliseconds have elapsed.
The reason that a batch is processed, even if the maximum batch size isnt reached, is to reduce the delay in processing and to continue to process events in a timely manner. This works well for pipelines that process a low volume of events.
Imagine that youve a pipeline that processes error logs from web servers and pushes them to Opensearch. Youre using Opensearch Dashboards to analyze the error logs. Because youre possibly dealing with a fairly low number of events, it might take a long time to reach 50 events. Logstash processes the events before reaching this threshold because otherwise there would be a long delay before we see the errors appear in Opensearch Dashboards.
Imagine that youve a pipeline that processes error logs from web servers and pushes them to OpenSearch. Youre using OpenSearch Dashboards to analyze the error logs. Because youre possibly dealing with a fairly low number of events, it might take a long time to reach 50 events. Logstash processes the events before reaching this threshold because otherwise there would be a long delay before we see the errors appear in OpenSearch Dashboards.
The default batch size and batch delay work for most cases. You dont need to change the default values unless you need to minutely optimize the performance.
@ -37,4 +37,4 @@ The default batch size and batch delay work for most cases. You dont need to
The number of pipeline workers are proportional to the number of CPU cores on the nodes.
If you have 5 workers running on a server with 2 CPU cores, the 5 workers won't be able to process events concurrently. On the other hand, running 5 workers on a server running 10 CPU cores limits the throughput of a Logstash instance.
Instead of running a fixed number of workers, which results in poor performance in some cases, Logstash examines the number of CPU cores of the instance and selects the number of pipeline workers to optimize its performance for the platform on which its running. For instance, your local development machine might now have the same processing power as a production server. So you don't need to manually configure Logstash for different machines.
Instead of running a fixed number of workers, which results in poor performance in some cases, Logstash examines the number of CPU cores of the instance and selects the number of pipeline workers to optimize its performance for the platform on which its running. For instance, your local development machine might not have the same processing power as a production server. So you don't need to manually configure Logstash for different machines.

View File

@ -8,9 +8,9 @@ has_toc: true
# Logstash
Logstash is a real-time event processing engine. It's part of the Opensearch stack which includes Opensearch, Beats, and Opensearch Dashboards.
Logstash is a real-time event processing engine. It's part of the OpenSearch stack which includes OpenSearch, Beats, and OpenSearch Dashboards.
You can send events to Logstash from many different sources. Logstash processes the events and sends it one or more destinations. For example, you can send access logs from a web server to Logstash. Logstash extracts useful information from each log and sends it to a destination like Opensearch.
You can send events to Logstash from many different sources. Logstash processes the events and sends it one or more destinations. For example, you can send access logs from a web server to Logstash. Logstash extracts useful information from each log and sends it to a destination like OpenSearch.
Sending events to Logstash lets you decouple event processing from your app. Your app only needs to send events to Logstash and doesnt need to know anything about what happens to the events afterwards.
@ -18,7 +18,7 @@ The open-source community originally built Logstash for processing log data but
## Structure of a pipeline
The way that Logstash works is that you configure a pipeline that has three phasesinputs, filters, and outputs.
The way that Logstash works is that you configure a pipeline that has three phases---inputs, filters, and outputs.
Each phase uses one or more plugins. Logstash has over 200 built-in plugins so chances are that youll find what you need. Apart from the built-in plugins, you can use plugins from the community or even write your own.
@ -42,7 +42,7 @@ where:
* `input` receives events like logs from multiple sources simultaneously. Logstash supports a number of input plugins for TCP/UDP, files, syslog, Microsoft Windows EventLogs, stdin, HTTP, and so on. You can also use an open source collection of input tools called Beats to gather events. The input plugin sends the events to a filter.
* `filter` parses and enriches the events in one way or the other. Logstash has a large collection of filter plugins that modify events and pass them on to an output. For example, a `grok` filter parses unstructured events into fields and a `mutate` filter changes fields. Filters are executed sequentially.
* `output` ships the filtered events to one or more destinations. Logstash supports a wide range of output plugins for destinations like Opensearch, TCP/UDP, emails, files, stdout, HTTP, Nagios, and so on.
* `output` ships the filtered events to one or more destinations. Logstash supports a wide range of output plugins for destinations like OpenSearch, TCP/UDP, emails, files, stdout, HTTP, Nagios, and so on.
Both the input and output phases support codecs to process events as they enter or exit the pipeline.
Some of the popular codecs are `json` and `multiline`. The `json` codec processes data thats in JSON format and the `multiline` codec merges multiple line events into a single line.
@ -53,7 +53,8 @@ You can also write conditional statements within pipeline configurations to perf
Make sure you have [Java Development Kit (JDK)](https://www.oracle.com/java/technologies/javase-downloads.html) version 8 or 11 installed.
1. Download the Logstash tarball from --.
1. Download the Logstash tarball from [OpenSearch downloads](https://opensearch.org/downloads.html).
2. Navigate to the downloaded folder in the terminal and extract the files:
```bash

View File

@ -7,9 +7,9 @@ nav_order: 220
# Send events to Opensearch
You can send Logstash events to an Opensearch cluster and then visualize your log data with Kibana.
You can send Logstash events to an Opensearch cluster and then visualize your events with Kibana.
Make sure you have Logstash, Opensearch, and Kibana installed.
Make sure you have [Logstash]({{site.url}}{{site.baseurl}}/logstash/index/#install-logstash-on-mac--linux), [OpenSearch]({{site.url}}{{site.baseurl}}/opensearch/install/index/), and [OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/install/index/).
{: .note }
## Opensearch output plugin