OpenSearch/shield/docs/public/clients/logstash.asciidoc

[[logstash]]
=== Shield with Logstash

IMPORTANT: Shield 2.0.x is compatible with Logstash 1.5 and above.

Logstash provides Elasticsearch https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html[output], https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html[input] and https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html[filter] plugins
used to index and retrieve documents through HTTP, transport or client node protocols.
All plugins support authentication and encryption over HTTP, while the output plugin additionally supports these
features over the transport protocol.

Note: When using the elasticsearch output, only the `transport` and `http` protocol are supported (i.e. `node` protocol is unsupported)

For information on setting up authentication and authorization on the Elasticsearch side, check the corresponding
documentation sections: <<authorization,_Authorization_>> and <<authentication,_Authentication_>>.

To configure the certificates and other SSL related options, see <<securing-nodes,_Securing Nodes_>>.

[[ls-user]]
==== Creating a user

By default, the Shield plugin installs a dedicated user <<roles,role>> that enables the creation of indices with names
that match the `logstash-*` regular expression, along with privileges to read, scroll, index, update, and delete 
documents on those indices:

[source,yaml]
--------------------------------------------------------------------------------------------
logstash:
  cluster: indices:admin/template/get, indices:admin/template/put
  indices:
    'logstash-*': indices:data/write/bulk, indices:data/write/delete, indices:data/write/update, indices:data/read/search, indices:data/read/scroll, create_index
--------------------------------------------------------------------------------------------

See the <<roles-file,_Role Definition File_>> section for information on modifying roles.

Create a user associated with the `logstash` role on the Elasticsearch cluster, using the <<esusers,`esusers` tool>>:

[source,shell]
--------------------------------------------------
esusers useradd <username> -p <password> -r logstash
--------------------------------------------------

NOTE: When using the transport protocol, the logstash user requires the predefined `transport_client` role in addition to the `logstash` role shown above (`-r logstash,transport_client`).

Once you've created the user, you are ready to configure Logstash.

[[ls-http]]
==== Connecting with HTTP/HTTPS

All three input, filter and output plugins support HTTP Basic Authentication as well as SSL/TLS.
The sections below demonstrate the output plugin's configuration parameters, but input and filter are the same.

[[ls-http-auth]]
===== Basic Authentication

To connect to an instance of Elasticsearch with Shield, set up the username and password credentials with the following 
configuration parameters:

[source, shell]
--------------------------------------------------
input { ... }
output {
  elasticsearch {
    protocol => "http"
    ...
    user => ... # string
    password => ... # string
  }
}
--------------------------------------------------

[[ls-http-ssl]]
===== SSL/TLS Configuration for HTTPS

To enable SSL/TLS encryption for HTTPS, use the following configuration block:

[source, shell]
--------------------------------------------------
input { ... }
output {
  elasticsearch {
    protocol => "http"
    ...
    ssl => true
    cacert => '/path/to/cert.pem' <1>
  }
}
--------------------------------------------------
<1> The path to the `.pem` file in your filesystem that contains the Certificate Authority's certificate.

[[ls-transport]]
==== Connecting with Transport protocol

By setting the "protocol" option to "transport", Logstash communicates with the Elasticsearch cluster through the same
protocol nodes use between each other. This avoids JSON un/marshalling and is therefore more efficient.

In order to unlock this option, it's necessary to install an additional plugin in Logstash using the following command:

[source, shell]
--------------------------------------------------
bin/plugin install logstash-output-elasticsearch-shield
--------------------------------------------------

[[ls-transport-auth]]
===== Authentication for Transport protocol

Transport protocol supports both basic auth and client-certificate authentication through the use of Public Key Infrastructure (PKI).

[[ls-transport-auth-basic]]
===== Basic Authentication

To connect to an instance of Elasticsearch with Shield using basic auth, set up the username and password credentials with the following configuration parameters:

[source, shell]
--------------------------------------------------
input { ... }
output {
  elasticsearch {
    protocol => "transport"
    ...
    user => ... # string
    password => ... # string
  }
}
--------------------------------------------------

[[ls-transport-auth-pki]]
===== PKI Authentication

To connect to an instance of Elasticsearch with Shield using client-certificate authentication you need to setup the keystore path which contain the client's certificate and the keystore password in the configuration:

[source, shell]
--------------------------------------------------
input { ... }
output {
  elasticsearch {
    protocol => "transport"    
    ...
    ssl => true
    keystore => ... # string
    keystore_password => ... # string
  }
}
--------------------------------------------------


[[ls-transport-conf]]
===== SSL Configuration for Transport or Node protocols

Specify the paths to the keystore and truststore `.jks` files with the following configuration parameters:

[source, shell]
--------------------------------------------------
input { ... }
output {
  elasticsearch {
    protocol => "transport"
    host => ... # string (optional)
    cluster => ... # string (optional)
    ...
    ssl => true
    keystore =>  ... # string
    keystore_password =>  ... # string
    truststore =>  ... # string
    truststore_password =>  ... # string
  }
}
--------------------------------------------------

For more information on encryption and certificates, see the <<ssl-tls,Securing Nodes>> section:

[[ls-failure]]
==== Failures

Logstash raises an exception that halts the processing pipeline when the server's certificate does not validate over SSL 
on any of the protocols discussed in this section. Same for the invalid user credentials.