diff --git a/_clients/logstash/advanced-config.md b/_clients/logstash/advanced-config.md new file mode 100644 index 00000000..d7da127c --- /dev/null +++ b/_clients/logstash/advanced-config.md @@ -0,0 +1,246 @@ +--- +layout: default +title: Advanced configurations +parent: Logstash +nav_order: 230 +--- + +# Advanced configurations + +This section describes how to set up advanced configuration options, like referencing field values and conditional statements, for Logstash. + +## Referencing field values + +To get access to a field, use the `- field` syntax. +You can also surround the field name by square brackets `- [field]` which makes it more explicit that you're referring to a field. + + +For example, if you have the following event: + +```bash +{ + "request": "/products/view/123", + "verb": "GET", + "response": 200, + "headers": { + "request_path" => "/" + } +} +``` + +To access the `request` field, use `- request` or `- [request]`. + +If you want to reference nested fields, use the square brackets syntax and specify the path to the field. With each level being enclosed within square brackets: `- [headers][request_path]`. + +You can reference fields using the `sprintf` format. This is also called string expansion. You need to add a % sign and then wrap the field reference within curly brackets. + +You need to reference field values when using conditional statements. + +For example, you can make the file name dynamic and contain the type of the processed events - either `access` or `error`. The `type` option is mainly used for conditionally applying filter plugins based on the type of events being processed. + +Let's add a `type` option and specify a value of `access`. + + +```yml +input { + file { + path => "" + start_position => "beginning" + type => "access" + } + http { + type => "access" + } +} + +filter { + mutate { + remove_field => {"host"} + } +} + +output { + stdout { + codec => rubydebug + } +file { + path => "%{[type]}.log" + } +} +``` + +Start Logstash and send an HTTP request. The processed event is output in the terminal. The event now includes a field named `type`. + +You'll see the `access.log` file created within the Logstash directory. + +## Conditional statements + +You can use conditional statements to control the flow of code execution based on some conditions. + +Syntax: + +```yml +if EXPR { + ... +} else if EXPR { + ... +} else { + ... +} +``` + +`EXPR` is any valid Logstash syntax that evaluates to a boolean value. +For example, you can check if an event type is set to `access` or `error` and perform some action based on that: + +```yml +if [type] == "access" { +... +} else if [type] == "error" { +file { .. } +} else { +... +} +``` + +You can compare a field value to some arbitrary value: + +```yml +if [headers][content_length] >= 1000 { +... +} +``` + +You can regex: + +```yml +if [some_field =~ /[0-9]+/ { + //some field only contains digits +} +``` + +You can use arrays: + +```yml +if [some_field] in ["one", "two", "three"] { + some field is either "one", "two", or "three" +} +``` + +You can use boolean operators: + +```yml +if [type] == "access" or [type] == "error" { + ... +} +``` + + +## Formatting dates + +You can use the `sprintf` format or string expansion to format dates. +For example, you might want the current date to be part of the filename. + +To format the date, add a plus sign in curly brackets followed by the date format - `%{+yyyy-MM-dd}`. + +```yml +file { + path => "%{[type]}_%{+yyyy_MM_dd}.log" +} +``` + +This is the date stored within the @timestamp fields, which is the time and date of the event. +Send a request to the pipeline and verify that a filename is outputted that contains the events date. + +You can embed the date in other outputs as well, for example into the index name in Opensearch. + +## Sending time information + +You can set the time of events. + +Logstash already sets the time when the event is received by the input plugin within the @timestamp field. +In some scenarios, you might need to use a different timestamp. +For example, if you have an eCommerce store and you process the orders daily at midnight. When Logstash receives the events at midnight, it sets the timestamp to the current time. +But you want it to be the time when the order is placed and not when Logstash received the event. + +Let's change the event timestamp to the date the request is received by the web server. You can do this using a filter plugin named `dates`. +The `dates` filter passes a `date` or `datetime` value from a field and uses the results as the event timestamp. + +Add the `date` plugin at the bottom of the `filter` block: + +```yml +date { + match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] +} +``` + +timestamp is the field that the `grok` pattern creates. +`Z` is the timezone. i.e., UTC offsets. + +Start Logstash and send an HTTP request. + +You can see that the filename contains the date of the request instead of the present date. + +If the passing of the date fails, the `filter` plugin adds a tag named `_datepassfailure` to the text field. + +After you have set the @timestamp field to a new value, you don't really need the other `timestamp` field anymore. You can remove it with the `remove_field` option. + +```yml +date { + match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] + remove_field => [ "timestamp" ] +} +``` + +## Parsing user agents + +The user agent is the last part of a log entry that consists of the name of the browser, the browser version, and the OS of the device. + +Users might be using a wide range of browsers, devices, and OS's. Doing this manually is hard. + +You can't use `grok` patterns because the `grok` pattern only matches the usage in the string as whole and doesn't figure out which browser the visitor used for instance. + +Logstash ships with a file containing regular expressions for this purpose. This makes it really easy to extract user agent information, which you could send to Opensearch and run aggregations on. + +To do this, add a `source` option that contains the name of the field. In this case, that's the `agent` field. +By default the user agent plugin, adds a number of fields at the top-level of the event. +Since that can get pretty confusing, we can add an option named `target` with a value of `ua`, short for user agent. What this does is that it nests the fields within an object named `ua`, making things more organized. + +```yml +useragent { + source => "agent" + target => "ua" +} +``` + +Start Logstah and send an HTTP request. + +You can see a field named `ua` with a number of keys including the browser name and version, the OS, and the device. + +You could Opensearch Dashboards to create a pie chart that shows how many visitors are from mobile devices and how many are desktop users. Or, you could get statistics on which browser versions are popular. + +## Enriching geographical data + +You can take an IP address and perform geographical lookup to resolve the geographical location of the user using the `geoip` filter. + +The `geoip` filter plugin ships with a database called `geolite 2`, which is provided by a company named MaxMind. `geolite 2` is a popular source of geographical data and it's available for free. +Add the `geoip` plugin at the bottom of the `else` block. + +The value of the `source` option is the name of the field containing the IP address, in this case that's `clientip`. You can make this field available using the `grok` pattern. + +```yml +geoip { + source => "clientip" +} +``` + +Start Logstash and send an HTTP request. + +Within the terminal, you see a new field named `geoip` that contains information such as the timezone, country, continent, city, postal code, and the latitude / longitude pair. + +If you only need the country name for instance, include an option named `fields` with an array of the field names that you want the `geoip` plugin to return. + +Some of the fields are not always available such as city name and region because translating IP addresses into geographical locations is generally not that accurate. If the `geoip` plugin fails to look up the geographical location, it adds a tag named `geoip_lookup_failure`. + +You can use the `geoip` plugin with the Opensearch output because `location` object within the `geoip` object, is a standard format for representing geospatial data in JSON. This is the same format as Opensearch uses for its `geo_point` data type. + +You can use the powerful geospatial queries of Opensearch for working with geographical data. diff --git a/_clients/logstash/common-filters.md b/_clients/logstash/common-filters.md new file mode 100644 index 00000000..48aacb48 --- /dev/null +++ b/_clients/logstash/common-filters.md @@ -0,0 +1,157 @@ +--- +layout: default +title: Common filter plugins +parent: Logstash +nav_order: 220 +--- + +# Common filter plugins + +This page contains a list of common filter plugins. + +## mutate + +You can use the `mutate` filter to change the data type of a field. For example, you can use the `mutate` filter if you're sending events to Opensearch and you need to change the data type of a field to match any existing mappings. + +To convert the `quantity` field from a `string` type to an `integer` type: + +```yml +input { + http { + host => "127.0.0.1" + port => 8080 + } +} + +filter { + mutate { + convert => {"quantity" => "integer"} + } +} + +output { + file { + path => "output.txt" + } +} +``` + +#### Sample output + +You can see that the type of the `quantity` field is changed from a `string` to an `integer`. + +```yml +{ + "quantity" => 3, + "host" => "127.0.0.1", + "@timestamp" => 2021-05-23T19:02:08.026Z, + "amount" => 10, + "@version" => "1", + "headers" => { + "request_path" => "/", + "connection" => "keep-alive", + "content_length" => "41", + "http_user_agent" => "PostmanRuntime/7.26.8", + "request_method" => "PUT", + "cache_control" => "no-cache", + "http_accept" => "*/*", + "content_type" => "application/json", + "http_version" => "HTTP/1.1", + "http_host" => "127.0.0.1:8080", + "accept_encoding" => "gzip, deflate, br", + "postman_token" => "ffd1cdcb-7a1d-4d63-90f8-0f2773069205" + } +} +``` + +Other data types you can convert to are `float`, `string`, and `boolean` values. If you pass in an array, the `mutate` filter converts all the elements in the array. If you pass a `string` like "world" to cast to an `integer` type, the result is 0 and Logstash continues processing events. + +Logstash supports a few common options for all filter plugins: + +Option | Description +:--- | :--- +`add_field` | Adds one or more fields to the event. +`remove_field` | Removes one or more events from the field. +`add_tag` | Adds one or more tags to the event. You can use tags to perform conditional processing on events depending on which tags they contain. +`remove_tag` | Removes one or more tags from the event. + +For example, you can remove the `host` field from the event: + +```yml +input { + http { + host => "127.0.0.1" + port => 8080 + } +} + +filter { + mutate { + remove_field => {"host"} + } +} + +output { + file { + path => "output.txt" + } +} +``` + +## grok + +With the `grok` filter, you can parse unstructured data and and structure it into fields. The `grok` filter uses text patterns to match text in your logs. You can think of text patterns as variables containing regular expressions. + +The format of a text pattern is as follows: + +```bash +%{SYNTAX:SEMANTIC} +``` + +`SYNTAX` is the format a piece of text should be in for the pattern to match. You can enter any of `grok`'s predefined patterns. For example, you can use the email identifier to match an email address from a given piece of text. + +`SEMANTIC` is an arbitrary name for the matched text. For example, if you're using the email identifier syntax, you can name it “email.” + +The following request consists of the IP address of the visitor, name of the visitor, the timestamp of the request, the HTTP verb and URL, the HTTP status code, and the number of bytes: + +```bash +184.252.108.229 - joe [20/Sep/2017:13:22:22 +0200] GET /products/view/123 200 12798 +``` + +To split this request into different fields: + +```yml +filter { + grok { + match => { "message" => " %{IP: ip_address} %{USER:identity} + %{USER:auth} \[%{HTTPDATE:reg_ts}\] + \"%{WORD:http_verb} + %{URIPATHPARAM: req_path} + \" %{INT:http_status:int} + %{INT:num_bytes:int}"} + } +} +``` + +where: + +- `IP`: matches the IP address field. +- `USER`: matches the user name. +- `WORD`: matches the HTTP verb. +- `URIPATHPARAM`: matches the URI path. +- `INT`: matches the HTTP status field. +- `INT`: matches the number of bytes. + +This is what the event looks like after the `grok` filter breaks it down into individual fields: + +```yml +ip_address: 184.252.108.229 +identity: joe +reg_ts: 20/Sep/2017:13:22:22 +0200 +http_verb:GET +req_path: /products/view/123 +http_status: 200 +num_bytes: 12798 +``` + +For common log formats, you use the predefined patterns defined here - [Logstash patterns](https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/ecs-v1). You cam make any adjustments to the results with the `mutate` filter. diff --git a/_clients/logstash/execution-model.md b/_clients/logstash/execution-model.md new file mode 100644 index 00000000..8c4cb7fc --- /dev/null +++ b/_clients/logstash/execution-model.md @@ -0,0 +1,40 @@ +--- +layout: default +title: Logstash execution model +parent: Logstash +nav_order: 210 +--- + +# Logstash execution model + +Here's a brief introduction to how Logstash processes events internally. + +## Handling events concurrently + +You can configure Logstash to have a number of inputs listening for events. Each input runs in its own thread to avoid inputs blocking each other. If you have two incoming events at the same time, Logstash handles both events concurrently. + +After receiving an event and possibly applying an input codec, Logstash sends the event to a work queue. Pipeline workers or batchers perform the rest of work involving filters and outputs along with any codec used at the output. Each pipeline worker also runs within its own thread meaning that Logstash processes multiple events simultaneously. + +## Processing events in batches + +A pipeline worker consumes events from the work queue in batches to optimize the throughput of the pipeline as a whole. + +One reason why Logstash works in batches is that some code needs to be executed regardless of how many events are processed at a time within the pipeline worker. Instead of executing that code 100 times for 100 events, it’s more efficient to execute it once for a batch of 100 events. + +Another reason is that some output plugins group together events as batches. For example, if you send 100 requests to Opensearch, the Opensearch output plugin uses the bulk API to send a single request that groups together the 100 requests. + +Logstash determines the batch size by two configuration options — a number representing the maximum batch size and the batch delay. The batch delay is how long Logstash waits before processing the unprocessed batch of events. +If you set the maximum batch size to 50 and the batch delay to 100 ms, Logstash processes a batch if they're either 50 unprocessed events in the work queue or if one hundred milliseconds have elapsed. + +The reason that a batch is processed, even if the maximum batch size isn’t reached, is to reduce the delay in processing and to continue to process events in a timely manner. This works well for pipelines that process a low volume of events. + +Imagine that you’ve a pipeline that processes error logs from web servers and pushes them to Opensearch. You’re using Opensearch Dashboards to analyze the error logs. Because you’re possibly dealing with a fairly low number of events, it might take a long time to reach 50 events. Logstash processes the events before reaching this threshold because otherwise there would be a long delay before we see the errors appear in Opensearch Dashboards. + +The default batch size and batch delay work for most cases. You don’t need to change the default values unless you need to minutely optimize the performance. + +## Optimizing based on CPU cores + +The number of pipeline workers are proportional to the number of CPU cores on the nodes. +If you have 5 workers running on a server with 2 CPU cores, the 5 workers won't be able to process events concurrently. On the other hand, running 5 workers on a server running 10 CPU cores limits the throughput of a Logstash instance. + +Instead of running a fixed number of workers, which results in poor performance in some cases, Logstash examines the number of CPU cores of the instance and selects the number of pipeline workers to optimize its performance for the platform on which its running. For instance, your local development machine might now have the same processing power as a production server. So you don't need to manually configure Logstash for different machines. diff --git a/_clients/logstash/index.md b/_clients/logstash/index.md new file mode 100644 index 00000000..945b5309 --- /dev/null +++ b/_clients/logstash/index.md @@ -0,0 +1,316 @@ +--- +layout: default +title: Logstash +nav_order: 200 +has_children: true +has_toc: true +--- + +# Logstash + +Logstash is a real-time event processing engine. It's part of the Opensearch stack which includes Opensearch, Beats, and Opensearch Dashboards. + +You can send events to Logstash from many different sources. Logstash processes the events and sends it one or more destinations. For example, you can send access logs from a web server to Logstash. Logstash extracts useful information from each log and sends it to a destination like Opensearch. + +Sending events to Logstash lets you decouple event processing from your app. Your app only needs to send events to Logstash and doesn’t need to know anything about what happens to the events afterwards. + +The open-source community originally built Logstash for processing log data but now you can process any of type of events, including events in XML or JSON format. + +## Structure of a pipeline + +The way that Logstash works is that you configure a pipeline that has three phases — inputs, filters, and outputs. + +Each phase uses one or more plugins. Logstash has over 200 built-in plugins so chances are that you’ll find what you need. Apart from the built-in plugins, you can use plugins from the community or even write your own. + +The structure of a pipeline is as follows: + +```yml +input { + input_plugin => {} +} + +filter { + filter_plugin => {} +} + +output { + output_plugin => {} +} +``` + +where: + +* `input` receives events like logs from multiple sources simultaneously. Logstash supports a number of input plugins for TCP/UDP, files, syslog, Microsoft Windows EventLogs, stdin, HTTP, and so on. You can also use an open source collection of input tools called Beats to gather events. The input plugin sends the events to a filter. +* `filter` parses and enriches the events in one way or the other. Logstash has a large collection of filter plugins that modify events and pass them on to an output. For example, a `grok` filter parses unstructured events into fields and a `mutate` filter changes fields. Filters are executed sequentially. +* `output` ships the filtered events to one or more destinations. Logstash supports a wide range of output plugins for destinations like Opensearch, TCP/UDP, emails, files, stdout, HTTP, Nagios, and so on. + +Both the input and output phases support codecs to process events as they enter or exit the pipeline. +Some of the popular codecs are `json` and `multiline`. The `json` codec processes data that’s in JSON format and the `multiline` codec merges multiple line events into a single line. + +You can also write conditional statements within pipeline configurations to perform certain actions, if a certain criteria is met. + +## Install Logstash on MAC / Linux + +Make sure you have [Java Development Kit (JDK)](https://www.oracle.com/java/technologies/javase-downloads.html) version 8 or 11 installed. + +1. Download the Logstash tarball from --. +2. Navigate to the downloaded folder in the terminal and extract the files: + + ```bash + tar -zxvf logstash-7.12.1-darwin-x86_64.tar.gz + ``` + +3. Navigate to the `logstash-7.12.1` directory. +- You can add your pipeline configurations to the `config` directory. Logstash saves any data from the plugins in the `data` directory. The `bin` directory contains the binaries for starting Logstash and managing plugins. + +## Process text from the terminal + +You can define a pipeline that listens for events on `stdin` and outputs events on `stdout`. `stdin` and `stdout` refer to the terminal in which you’re running Logstash. + +To enter some text in the terminal and see the event data in the output: + +1. Use the `-e` argument to pass a pipeline configuration directly to the Logstash binary. In this case, `stdin` is the input plugin and `stdout` is the output plugin: + + ```bash + bin/logstash -e "input { stdin { } } output { stdout { } }" + ``` + Add the `—debug` flag to see a more detailed output. + +2. Enter "hello world" in your terminal. Logstash processes the text and outputs it back to the terminal: + + ```yml + { + "message" => "hello world", + "host" => "a483e711a548.ant.amazon.com", + "@timestamp" => 2021-05-30T05:15:56.816Z, + "@version" => "1" + } + ``` + + The `message` field contains your raw input. The `host` field is an IP address when you don’t run Logstash locally. `@timestamp` shows the date and time for when the event is processed. Logstash uses the `@version` field for internal processing. + +3. Press `Ctrl + C` to shut down Logstash. + +### Troubleshooting + +If you already have a Logstash process running, you’ll get an error. To fix this issue: + +1. Delete the `.lock` file from the `data` directory: + + ```bash + cd data + rm -rf .lock + ``` + +2. Restart Logstash. + +## Process JSON or HTTP input and output it to a file + +To define a pipeline that handles JSON requests: + +1. Open the `config/pipeline.conf` file in any text editor you like. You can create a pipeline configuration file with any extension, the `.conf` extension is a Logstash convention. Add the `json` codec to accept JSON as the input and the `file` plugin to output the processed events to a `.txt` file: + + ```yml + input { + stdin { + codec => json + } + } + output { + file { + path => "output.txt" + } + } + ``` + + To process inputs from a file, add an input file to the `events-data` directory and then pass its path to the `file` plugin at the input: + + ```yml + input { + file { + path => "events-data/input_data.log" + } + } + ``` + +2. Start Logstash: + + ```bash + $ bin/logstash -f config/pipeline.conf + ``` + + `config/pipeline.conf` is a relative path to the `pipeline.conf` file. You can use an absolute path as well. + +3. Add a JSON object in the terminal: + + ```json + { "amount": 10, "quantity": 2} + ``` + + The pipeline only handles a single line of input. If you paste some JSON that spans multiple lines, you’ll get an error. + +4. Check that the fields from the JSON object are added to the `output.txt` file: + + ```json + $ cat output.txt + + { + "@version": "1", + "@timestamp": "2021-05-30T05:52:52.421Z", + "host": "a483e711a548.ant.amazon.com", + "amount": 10, + "quantity": 2 + } + ``` + + If you type in some invalid JSON as the input, you'll see a JSON parsing error. Logstash doesn't discard the invalid JSON because you still might want to do something with it. For example, you can trigger an email or send a notification to a Slack channel. + +To define a pipeline that handles HTTP requests: + +1. Use the `http` plugin to send events to Logstash through HTTP: + + ```yml + input { + http { + host => "127.0.0.1" + port => 8080 + } + } + + output { + file { + path => "output.txt" + } + } + ``` + + If you don’t specify any options, the `http` plugin binds to `localhost` and listens on port 8080. + +2. Start Logstash: + + ```bash + $ bin/logstash -f config/pipeline.conf + ``` + +3. Use Postman to send an HTTP request. Set `Content-Type` to an HTTP header with a value of `application/json`: + + ```json + PUT 127.0.0.1:8080 + + { + "amount": 10, + "quantity": 2 + } + ``` + + Or, you can use the `curl` command: + + ```bash + curl -XPUT -H "Content-Type: application/json" -d ' {"amount": 7, "quantity": 3 }' http://localhost:8080 (http://localhost:8080/) + ``` + + Even though we haven't added the `json` plugin to the input, the pipeline configuration still works because the HTTP plugin automatically applies the appropriate codec based on the `Content-Type` header. + If you specify a value of `applications/json`, Logstash parses the request body as JSON. + + The `headers` field contains the HTTP headers that Logstash receives: + + ```json + { + "host": "127.0.0.1", + "quantity": "3", + "amount": 10, + "@timestamp": "2021-05-30T06:05:48.135Z", + "headers": { + "http_version": "HTTP/1.1", + "request_method": "PUT", + "http_user_agent": "PostmanRuntime/7.26.8", + "connection": "keep-alive", + "postman_token": "c6cd29cf-1b37-4420-8db3-9faec66b9e7e", + "http_host": "127.0.0.1:8080", + "cache_control": "no-cache", + "request_path": "/", + "content_type": "application/json", + "http_accept": "*/*", + "content_length": "41", + "accept_encoding": "gzip, deflate, br" + }, + "@version": "1" + } + ``` + + +## Automatically reload the pipeline configuration + +You can configure Logstash to detect any changes to the pipeline configuration file or the input log file and automatically reload the configuration. + +The `stdin` plugin doesn’t supporting automatic reloading. +{: .note } + +1. Add an option named `start_position` with a value of `beginning` to the input plugin: + + ```yml + input { + file { + path => "/Users//Desktop/logstash7-12.1/events-data/input_file.log" + start_position => "beginning" + } + } + ``` + + Logstash only processes any new events added to the input file and ignores the ones that it has already processed to avoid processing the same event more than once on restart. + + Logstash records its progress in a file that's referred to as a `sinceDB` file. Logstash creates a `sinceDB` file for each file that it watches for changes. + +2. Open the `sinceDB` file to check how much of the input files are processed: + + ```bash + cd data/plugins/inputs/file/ + ls -al + + -rw-r--r-- 1 user staff 0 Jun 13 10:50 .sincedb_9e484f2a9e6c0d1bdfe6f23ac107ffc5 + + cat .sincedb_9e484f2a9e6c0d1bdfe6f23ac107ffc5 + + 51575938 1 4 7727 + ``` + + The last number in the `sinceDB` file (7727) is the byte offset of the last known event processed. + +5. To process the input file from the beginning, delete the `sinceDB` file: + + ```yml + rm .sincedb_* + ``` + +2. Start Logstash with a `—-config.reload.automatic` argument: + + ```bash + bin/logstash -f config/pipeline.conf --config.reload.automatic + ``` + + The `reload` option only reloads if you add a new line at the end of the pipeline configuration file. + + Sample output: + + ```yml + { + "message" => "216.243.171.38 - - [20/Sep/2017:19:11:52 +0200] \"GET /products/view/123 HTTP/1.1\" 200 12798 \"https://codingexplained.com/products\" \"Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)\"", + "@version" => "1", + "host" => "a483e711a548.ant.amazon.com", + "path" => "/Users/kumarjao/Desktop/odfe1/logstash-7.12.1/events-data/input_file.log", + "@timestamp" => 2021-06-13T18:03:30.423Z + } + { + "message" => "91.59.108.75 - - [20/Sep/2017:20:11:43 +0200] \"GET /js/main.js HTTP/1.1\" 200 588 \"https://codingexplained.com/products/view/863\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0\"", + "@version" => "1", + "host" => "a483e711a548.ant.amazon.com", + "path" => "/Users/kumarjao/Desktop/odfe1/logstash-7.12.1/events-data/input_file.log", + "@timestamp" => 2021-06-13T18:03:30.424Z + } + ``` + +7. Add a new line to the input file. + - Logstash immediately detects the change and processes the new line as an event. + +8. Make a change to the `pipeline.conf` file. + - Logstash immediately detects the change and reloads the modified pipeline.