203 lines
7.0 KiB
Plaintext
203 lines
7.0 KiB
Plaintext
[[logstash]]
|
||
=== Using Logstash with Shield
|
||
|
||
IMPORTANT: Shield 1.0+ is compatible with Logstash 1.5 and above.
|
||
|
||
Logstash provides Elasticsearch https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html[output], https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html[input] and https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html[filter] plugins
|
||
used to index and retrieve documents through HTTP, transport or client node protocols.
|
||
All plugins support authentication and encryption over HTTP, while the output plugin additionally supports these
|
||
features over the transport protocol.
|
||
|
||
NOTE: When using the `elasticsearch` output, only the `transport` and `http` protocol are supported (i.e. `node` protocol is unsupported)
|
||
|
||
[float]
|
||
[[ls-user]]
|
||
==== Creating a user
|
||
|
||
By default, the Shield plugin installs a dedicated user <<roles,role>> that enables the creation of indices with names
|
||
that match the `logstash-*` regular expression, along with privileges to read, scroll, index, update, and delete
|
||
documents on those indices:
|
||
|
||
[source,yaml]
|
||
--------------------------------------------------------------------------------------------
|
||
logstash:
|
||
cluster: indices:admin/template/get, indices:admin/template/put
|
||
indices:
|
||
'logstash-*': indices:data/write/bulk, indices:data/write/delete, indices:data/write/update, indices:data/read/search, indices:data/read/scroll, create_index
|
||
--------------------------------------------------------------------------------------------
|
||
|
||
See the <<defining-roles,_Role Definition File_>> section for information on modifying roles.
|
||
|
||
Create a user associated with the `logstash` role on the Elasticsearch cluster, using the <<esusers,`esusers` tool>>:
|
||
|
||
[source,shell]
|
||
--------------------------------------------------
|
||
esusers useradd <username> -p <password> -r logstash
|
||
--------------------------------------------------
|
||
|
||
NOTE: When using the transport protocol, the logstash user requires the predefined `transport_client` role in addition to the `logstash` role shown above (`-r logstash,transport_client`).
|
||
|
||
Once you've created the user, you are ready to configure Logstash.
|
||
|
||
[float]
|
||
[[ls-http]]
|
||
==== Connecting with HTTP/HTTPS
|
||
|
||
When you set the `protocol` option to `http`, Logstash communicates with the Elasticsearch cluster through the REST APIs over HTTP.
|
||
|
||
[float]
|
||
[[ls-http-auth]]
|
||
===== Authentication for HTTP protocol
|
||
|
||
HTTP protocol supports both basic auth and client-certificate authentication through the use of Public Key Infrastructure (PKI).
|
||
|
||
[float]
|
||
[[ls-http-auth-basic]]
|
||
===== Basic Authentication
|
||
|
||
The input, filter, and output plugins all support HTTP Basic Authentication. To use basic authentication when connecting to an instance of Elasticsearch with Shield, you configure the plugins to include username and password credentials with each request. For example, the following snippet configures credentials for the output plugin. The credentials are configured the same way for each plugin type.
|
||
|
||
[source, shell]
|
||
--------------------------------------------------
|
||
input { ... }
|
||
output {
|
||
elasticsearch {
|
||
protocol => "http"
|
||
...
|
||
user => ... # string
|
||
password => ... # string
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
|
||
[float]
|
||
[[ls-http-auth-pki]]
|
||
===== PKI Authentication
|
||
|
||
Elasticsearch Output version 1.0.1 onwards supports the use of X.509 client-certificate to authenticate Logstash requests. To enable this you need to set up the following configuration parameters:
|
||
|
||
[source, shell]
|
||
--------------------------------------------------
|
||
input { ... }
|
||
output {
|
||
elasticsearch {
|
||
protocol => "http"
|
||
...
|
||
keystore => ... # string
|
||
keystore_password => ... # string
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
|
||
[float]
|
||
[[ls-http-ssl]]
|
||
===== SSL/TLS Configuration for HTTPS
|
||
|
||
To enable SSL/TLS encryption for HTTPS, use the following configuration block:
|
||
|
||
[source, shell]
|
||
--------------------------------------------------
|
||
input { ... }
|
||
output {
|
||
elasticsearch {
|
||
protocol => "http"
|
||
...
|
||
ssl => true
|
||
cacert => '/path/to/cert.pem' <1>
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
<1> The path to the `.pem` file in your filesystem that contains the Certificate Authority's certificate.
|
||
|
||
[float]
|
||
[[ls-transport]]
|
||
==== Connecting with Transport protocol
|
||
|
||
When you set the `protocol` option to `transport`, Logstash communicates with the Elasticsearch cluster through the same
|
||
protocol nodes use between each other. This avoids JSON un/marshalling and is therefore more efficient.
|
||
|
||
In order to unlock this option, it's necessary to install an additional plugin in Logstash using the following command:
|
||
|
||
[source, shell]
|
||
--------------------------------------------------
|
||
bin/plugin install logstash-output-elasticsearch-shield
|
||
--------------------------------------------------
|
||
|
||
[float]
|
||
[[ls-transport-auth]]
|
||
===== Authentication for Transport protocol
|
||
|
||
Transport protocol supports both basic auth and client-certificate authentication through the use of Public Key Infrastructure (PKI).
|
||
|
||
[float]
|
||
[[ls-transport-auth-basic]]
|
||
===== Basic Authentication
|
||
|
||
To connect to an instance of Elasticsearch with Shield using basic auth, set up the username and password credentials with the following configuration parameters:
|
||
|
||
[source, shell]
|
||
--------------------------------------------------
|
||
input { ... }
|
||
output {
|
||
elasticsearch {
|
||
protocol => "transport"
|
||
...
|
||
user => ... # string
|
||
password => ... # string
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
|
||
[float]
|
||
[[ls-transport-auth-pki]]
|
||
===== PKI Authentication
|
||
|
||
To connect to an instance of Elasticsearch with Shield using client-certificate authentication you need to setup the keystore path which contain the client's certificate and the keystore password in the configuration:
|
||
|
||
[source, shell]
|
||
--------------------------------------------------
|
||
input { ... }
|
||
output {
|
||
elasticsearch {
|
||
protocol => "transport"
|
||
...
|
||
ssl => true
|
||
keystore => ... # string
|
||
keystore_password => ... # string
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
|
||
[float]
|
||
[[ls-transport-conf]]
|
||
===== SSL Configuration for Transport or Node protocols
|
||
|
||
Specify the paths to the keystore and truststore `.jks` files with the following configuration parameters:
|
||
|
||
[source, shell]
|
||
--------------------------------------------------
|
||
input { ... }
|
||
output {
|
||
elasticsearch {
|
||
protocol => "transport"
|
||
host => ... # string (optional)
|
||
cluster => ... # string (optional)
|
||
...
|
||
ssl => true
|
||
keystore => ... # string
|
||
keystore_password => ... # string
|
||
truststore => ... # string
|
||
truststore_password => ... # string
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
|
||
For more information on encryption and certificates, see the <<ssl-tls,Securing Nodes>> section:
|
||
|
||
[float]
|
||
[[ls-failure]]
|
||
==== Failures
|
||
|
||
Logstash raises an exception that halts the processing pipeline when the server's certificate does not validate over SSL
|
||
on any of the protocols discussed in this section. Same for the invalid user credentials.
|