Shield Docs: Updated Marvel config and license management for 2.0.

Original commit: elastic/x-pack-elasticsearch@3d225c2562
This commit is contained in:
debadair 2015-10-15 18:32:57 -07:00
parent aeffd696ff
commit 0123d33e87
4 changed files with 165 additions and 189 deletions

View File

@ -12,6 +12,6 @@ include::configuring-clients-integrations/hadoop.asciidoc[]
include::configuring-clients-integrations/logstash.asciidoc[] include::configuring-clients-integrations/logstash.asciidoc[]
include::configuring-clients-integrations/marvel.asciidoc[]
include::configuring-clients-integrations/kibana.asciidoc[] include::configuring-clients-integrations/kibana.asciidoc[]
include::configuring-clients-integrations/marvel.asciidoc[]

View File

@ -96,16 +96,6 @@ kibana3_monitoring:
- "cn=Jack Nicholson,dc=example,dc=com" - "cn=Jack Nicholson,dc=example,dc=com"
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
** If you are using a PKI realm, you assign roles on a per user basis in the role mapping file.
For example, the following snippet assigns the `kibana3_monitoring` role to the users named `Jack Nicholson` and `Robert De Niro.
+
[source,yaml]
--------------------------------------------------------------------------------
kibana3_monitoring:
- "cn=Jack Nicholson,ou=example,o=com"
- "cn=Robert De Niro,ou=example,o=com"
--------------------------------------------------------------------------------
. If you have <<ssl-tls, enabled SSL encryption>> in Shield and are using your own Certificate Authority (CA) to sign certificates for your nodes, configure your browser or operating system to trust your CA. When you access Kibana, your browser verifies that the certificate received from the Elasticsearch node is trusted before sending a request to the node. Establishing this trust requires that either your browser or operating system trust the CA that signed the node's certificate. Consult your local IT professional for information about the recommended procedure for adding trusted CAs in your organization. . If you have <<ssl-tls, enabled SSL encryption>> in Shield and are using your own Certificate Authority (CA) to sign certificates for your nodes, configure your browser or operating system to trust your CA. When you access Kibana, your browser verifies that the certificate received from the Elasticsearch node is trusted before sending a request to the node. Establishing this trust requires that either your browser or operating system trust the CA that signed the node's certificate. Consult your local IT professional for information about the recommended procedure for adding trusted CAs in your organization.
. Access Kibana 3 from your browser and verify that you can sign in as a user. For example, you could log in as the `jacknich` user created in step 4. . Access Kibana 3 from your browser and verify that you can sign in as a user. For example, you could log in as the `jacknich` user created in step 4.
@ -217,16 +207,6 @@ kibana4_monitoring:
- "cn=Jack Nicholson,dc=example,dc=com" - "cn=Jack Nicholson,dc=example,dc=com"
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
** If you are using a PKI realm, you assign roles on a per user basis in the role mapping file.
For example, the following snippet assigns the `kibana4_monitoring` role to the users named `Jack Nicholson` and `Robert De Niro`.
+
[source,yaml]
--------------------------------------------------------------------------------
kibana4_monitoring:
- "cn=Jack Nicholson,ou=example,o=com"
- "cn=Robert De Niro,ou=example,o=com"
--------------------------------------------------------------------------------
. If you have enabled SSL encryption in Shield, configure Kibana 4 to connect to Elasticsearch via HTTPS. To do this: . If you have enabled SSL encryption in Shield, configure Kibana 4 to connect to Elasticsearch via HTTPS. To do this:
.. Specify the HTTPS protocol in the `elasticsearch.url` setting in the Kibana configuration file, `kibana.yml`: .. Specify the HTTPS protocol in the `elasticsearch.url` setting in the Kibana configuration file, `kibana.yml`:

View File

@ -1,124 +1,115 @@
[[marvel]] [[marvel]]
=== Using Marvel with Shield === Using Marvel with Shield
Marvel consists of a user interface over a data exporter known as the _agent_. The agent runs on each node and accesses https://www.elastic.co/guide/en/marvel/current/introduction.html[Marvel] consists of two
that node's monitoring API. The agent can store this collected data locally, on the cluster, or send the data to an components: a Marvel agent that you install on on each node in your cluster, and a Marvel application you install in https://www.elastic.co/guide/en/kibana/current/introduction.html[Kibana]. The Marvel agent collects and indexes metrics from Elasticsearch and you visualize the data through the Marvel dashboards in Kibana. The agent can index data on the same cluster, or send it to an external monitoring cluster.
external monitoring cluster. Users can view and analyze the collected data with the Marvel UI.
To work with the Shield plugin, Marvel's configuration needs to be adapted for the _production_ cluster, which is the To use Marvel with Shield enabled, you need to <<kibana, set up Kibana to work with Shield>> and create at least one user for the Marvel app. If you are using an external monitoring cluster, you also need to configure a user for the Marvel agent and configure the agent to use the appropriate
cluster being monitored, as well as the _monitoring_ cluster, where the monitoring data is stored. For clusters that credentials when communicating with the monitoring cluster.
store their own monitoring data, apply both sets of settings to the single, production cluster.
You will configure at least two users to work with Marvel. These users have to exist on the monitoring cluster.
* The Agent needs to be assigned a user with the correct <<roles,privileges>> to write data to the Marvel indices
named `.marvel-*`, check the Marvel index template, and upload the Marvel index template. You need only one agent user.
* Marvel UI users must authenticate and have privileges to read data from the Marvel indices. These users also
need to able to call the Nodes Info API in order to get the monitoring cluster's Elasticsearch version.
This version check allows Marvel to be compatible with many versions of Elasticsearch. You can have as many of
these end users configured as you would like.
The default `roles.yml` file includes definitions for these two roles. The steps below show you how to create these
users on the monitoring cluster.
[float] [float]
[[monitoring-cluster]] [[marvel-app-users]]
==== Monitoring Cluster Settings ==== Setting Up Marvel App Users
The monitoring cluster is used to both store and view the Marvel data. When configuring Shield, you need to perform the When Shield is enabled, Kibana users are prompted to log in when they access the UI. To use
following actions: the Marvel app, a user must have access to the Kibana indices and permission to read from the
Marvel indices.
* Make sure there is a user created with the `marvel_agent` role. Marvel uses this to export the data. You set up Marvel app users on the cluster where the monitoring data is being stored. To grant
* Make sure there is a user created with the `marvel_user` role. You use this to view the Marvel UI and get license information. all of the necessary permissions, assign user the `kibana_user` and `marvel_user` roles defined
* When using Marvel on a production server, you must enter your Marvel License. This license is stored in the in `roles.yml`:
monitoring cluster. This step needs to be performed once, by a user with permissions to write to the `.marvel-kibana`
index. The .marvel-kibana index is used to store Marvel UI settings (for example, set custom warning levels) and
therefore write permission for `.marvel-kibana` is required for UI customizations. Both storing license and storing
settings can be done by any user added to the marvel_user role.
This is in the default `roles.yml` * If you're using the default `esusers` realm, you can assign roles when you <<esusers-add, add a
user>>, or modify the role assignments with the <<esusers-roles, `roles`>> command. For example,
the following command creates a user named `jacknich` and assigns the `marvel_user` role:
+
[source,console]
--------------------------------------------------------------------------------
esusers useradd jacknich -r kibana_user,marvel_user -p password
--------------------------------------------------------------------------------
* If you are using an LDAP or Active Directory realm, you can either assign roles on a per user
basis, or assign roles to groups of users. By default, role mappings are configured in
<<mapping-roles, `config/shield/role_mapping.yml`>>. For example, the following snippet assigns
the user named Jack Nicholson to the `kibana_user` and `marvel_user` roles:
+
[source,yaml]
--------------------------------------------------------------------------------
kibana_user:
- "cn=Jack Nicholson,dc=example,dc=com"
marvel_user:
- "cn=Jack Nicholson,dc=example,dc=com"
--------------------------------------------------------------------------------
[float]
[[configuring-marvel-agent-shield]]
==== Configuring Marvel Agent to Communicate with a Shield-Enabled Monitoring Cluster
To configure the Marvel agent to communicate with a secured monitoring cluster:
. Configure a user on the monitoring cluster who has the `marvel_agent` role, which is defined in
`roles.yml`. For example:
+
[source,console]
--------------------------------------------------------------------------------
esusers useradd agent-user -r marvel_agent -p password
--------------------------------------------------------------------------------
+
.Marvel Agent Role
[source,yaml] [source,yaml]
-------------------------------------------------- --------------------------------------------------
marvel_agent: marvel_agent:
cluster: indices:admin/template/get, indices:admin/template/put cluster: indices:admin/template/get, indices:admin/template/put
indices: indices:
'.marvel-*': indices:data/write/bulk, create_index '.marvel-*': indices:data/write/bulk, create_index
marvel_user:
cluster: cluster:monitor/nodes/info, cluster:admin/plugin/license/get
indices:
'.marvel-*': all
-------------------------------------------------- --------------------------------------------------
Once the roles are configured, create a user for the agent: . On each node in the cluster being monitored, configure a Marvel HTTP exporter
in `elasticsearch.yml` and resart Elasticsearch. In the exporter configuration,
you need to:
+
--
.. Set the `type` to `http`.
.. Specify the location of the monitoring cluster in the `host` setting.
.. Provide the agent user credentials with the `username` and `password` settings.
[source,shell] For example:
--------------------------------------------------
bin/shield/esusers useradd marvel_export -p strongpassword -r marvel_agent
--------------------------------------------------
Then create one or more users for the Marvel UI:
[source,shell]
--------------------------------------------------
bin/shield/esusers useradd USER -p strongerpassword -r marvel_user
--------------------------------------------------
[float]
==== Production Cluster Settings
The Marvel agent is installed on every node in the production cluster. The agent collects monitoring data from the
production cluster and stores the data on the monitoring cluster. The agent's configuration specifies a list of
hostname and port combinations for access to the monitoring cluster.
When the monitoring cluster uses the Shield plugin and is configured to accept only HTTPS requests, you must configure the agent
on the production cluster to use HTTPS instead of the default HTTP protocol.
Authentication and protocol configuration are both controlled by the `marvel.agent.exporter.es.hosts` setting in the
node's `elasticsearch.yml` file. The setting accepts a list of monitoring cluster servers to serve as a fallback
in case a server is unavailable. Each of these servers must be properly configured, as in the following example:
Example `marvel.agent.exporter.es.hosts` setting
[source,yaml]
-------------------------------------------------------------------------------------------------------------------
marvel.agent.exporter.es.hosts: [ "https<1>://USER:PASSWORD<2>@node01:9200", "https://USER:PASSWORD@node02:9200"]
-------------------------------------------------------------------------------------------------------------------
<1> Indicates to use HTTPS.
<2> Username and password. The user needs to be configured on the Monitoring Cluster as described in the next section.
When the monitoring cluster uses HTTPS, the Marvel agent will attempt to validate the certificate of the Elasticsearch
node in the monitoring cluster. If you are using your own CA you should specify a trust store that has the signing
certificate of the CA. Here is an example config for the `marvel.agent.exporter.es.truststore.*` settings:
[source,yaml] [source,yaml]
------------------------------------------------------------------------------------------------------------- --------------------------------------------------
marvel.agent.exporter.es.hosts: [ "https://USER:PASSWORD@node01:9200", "https://USER:PASSWORD@node02:9200"] marvel.agent.exporters:
marvel.agent.exporter.es.ssl.truststore.path: FULL_FILE_PATH id1:
marvel.agent.exporter.es.ssl.truststore.password: PASSWORD type: http
------------------------------------------------------------------------------------------------------------- host: [http://es-mon1:9200, http://es-mon2:9200]
auth:
username: agent-user
password: password
--------------------------------------------------
See the http://www.elastic.co/guide/en/marvel/current/configuration.html[Marvel documentation] for more details about If SSL/TLS is enabled on the monitoring cluster:
other SSL related settings.
NOTE: The 1.3.0 release of Marvel adds HTTPS support. .. Specify the HTTPS protocol when setting the monitoring server host.
.. Specify a truststore that contains the CA certificate to use to verify the identities of the
nodes in the monitoring cluster. You need to set the `truststore.path` and
`truststore.password`.
[float] For example:
==== Marvel user interface & Sense
The Marvel UI supports SSL without the need for any additional configuration. You can change URL access scheme for Marvel to [source,yaml]
HTTPS. --------------------------------------------------
marvel.agent.exporters:
id1:
type: http
host: [ https://es-mon1:9200, https://es-mon2:9200]
Users attempting to access the Marvel UI with the URL `https://HOST:9200/_plugin/marvel` must provide valid auth:
credentials. See <<monitoring-cluster,Monitoring Cluster settings>> for information on the required user configuration. username: agent-user
password: password
Sense also supports HTTPS access. Users that access Sense over URLs of the form ssl:
`https://host:9200/_plugin/marvel/sense/index.html` must provide valid credentials if they have not already truststore.path: /path/to/file
authenticated to a dashboard. truststore.password: password
id2:
Users connecting to the production cluster with Sense must provide valid credentials. Clusters must be configured to type: local
enable cross-origin requests to enable users to connect with Sense. See the <<cors, CORS>> documentation for details. --------------------------------------------------
--
NOTE: Providing user credentials to Sense in order to access another cluster is only supported in releases 1.3.0 and
later of Marvel.

View File

@ -1,109 +1,114 @@
[[license-management]] [[license-management]]
== Managing Shield Licenses == Managing Your License
When installing Shield for the first time, having the license plugin installed is the minimum required for Shield to work. When you initially install Shield, a 30 day trial license is installed that allows access to all features. At the end of the trial period, you can https://www.elastic.co/subscriptions/shield[purchase a subscription] to keep using the full functionality of Shield along with Marvel and Watcher.
You can just start up the node and everything will just work as expected. The first time you start up the node, a 30 days
trial license will automatically be created which will enable Shield to be fully operational. Within these 30 days, you IMPORTANT: When your license expires, Shield operates in a degraded mode where access to the Elasticsearch cluster health, cluster stats, and index stats APIs is blocked. Shield keeps on protecting your cluster, but you won't be able to monitor its operation until you update your license. For more information, see <<license-expiration, License Expiration>>.
will be able to replace the trial license with another one that will be provided to you up on purchase.
[float] [float]
[[installing-license]] [[installing-license]]
=== Updating Your Shield License === Updating Your License
Updating the
license can be done at runtime (no need to shutdown the nodes) using a dedicated API.
IMPORTANT: With a valid license, Shield will be fully operational. Upon license expiry, Shield will operate in a You can update your license at runtime without shutting down your nodes. License updates take
degraded mode, where cluster health, cluster stats, and index stats APIs will be blocked. All other operations will effect immediately. The license is provided as a _JSON_ file that you install with the `license`
continue operating normally. Find out more about <<license-expiration, Shield license expiration>>. API. You need cluster admin privileges to install the license.
To update your license:
The license itself is a _JSON_ file containing all information about the license (e.g. feature name, expiry date, etc...). . Send a request to the `license` API and specify the file that contains your new license:
To install or update the license use the following REST API: +
[source,shell] [source,shell]
----------------------------------------------------------------------- -----------------------------------------------------------------------
curl -XPUT -u admin 'http://<host>:<port>/_license' -d @license.json curl -XPUT -u admin 'http://<host>:<port>/_license' -d @license.json
----------------------------------------------------------------------- -----------------------------------------------------------------------
+
Where: Where:
+
* `<host>` is the hostname of the Elasticsearch node (`localhost` if executing locally) * `<host>` is the hostname of the Elasticsearch node (`localhost` if executing locally)
* `<port>` is the http port (defaults to `9200`) * `<port>` is the http port (defaults to `9200`)
* `license.json` is the license json file * `license.json` is the license JSON file
NOTE: The put license API is protected under the cluster admin privilege, therefore it has to be executed . If the license you are installing does not support all of the features available with your
by a user with the appropriate permissions. previous license, you will be notified in the response. To complete the license installation,
you must resubmit the license update request and set the `acknowledge` parameter to `true` to
indicate that you are aware of the changes.
+
[source,shell]
-----------------------------------------------------------------------
curl -XPUT -u admin 'http://<host>:<port>/_license?acknowledge=true' -d @license.json
-----------------------------------------------------------------------
[float] [float]
[[listing-licenses]] [[listing-licenses]]
=== Listing Installed Licenses === Viewing the Installed License
You can list all currently installed licenses by executing the following REST API: You can also use the `license` API to retrieve the currently installed license:
[source,shell] [source,shell]
----------------------------------------------------- -----------------------------------------------------
curl -XGET -u admin:password 'http://<host>:<port>/_license' curl -XGET -u admin:password 'http://<host>:<port>/_license'
-----------------------------------------------------
The response of this command will be a JSON listing all available licenses. In the case of Shield, the following
entry will be shown:
[source,json]
--------------------------------------------
{ {
licenses: [ "license" : {
... "status" : "active",
{ "uid" : "0a98411f-73f4-4c67-954c-724874ed5488",
status: "active", "type" : "trial",
uid: "sample_uid", "issue_date" : "2015-10-13T18:18:20.709Z",
type: "sample_type", "issue_date_in_millis" : 1444760300709,
subscription_type: "sample_subscription_type", "expiry_date" : "2015-11-12T18:18:20.709Z",
"issue_date" : "2015-01-26T00:00:00.000Z", "expiry_date_in_millis" : 1447352300709,
"issue_date_in_millis" : 1422230400000, "max_nodes" : 1000,
feature: "shield", "issued_to" : "elasticsearch",
"expiry_date" : "2015-04-26T23:59:59.999Z", "issuer" : "elasticsearch"
"expiry_date_in_millis" : 1430092799999,
max_nodes: 1,
issued_to: "sample customer",
issuer: "elasticsearch"
} }
...
]
} }
-------------------------------------------- --------------------------------------------
NOTE: The get license API is protected under the cluster admin privilege, therefore it has to be executed NOTE: You need cluster admin privileges to retrieve the license.
by a user with the appropriate permissions.
[float] [float]
[[license-expiration]] [[license-expiration]]
=== License Expiration === License Expiration
License expiration should never be a surprise. Beginning 30 days from license expiration, Shield will begin logging daily messages License expiration should never be a surprise. If you're using Marvel, a license expiration
containing the license expiration date and a brief description of unlicensed behavior. Beginning 7 days from license expiration, warning is displayed prominently if your license expires within 30 days. Warnings are
Shield will begin logging error messages every 10 minutes with the same information. After expiration, Shield will continue to also displayed on startup and written to the Elasticsearch log starting 30 days from the expiration date. These error messages tell you when the license expires and what features will be disabled if
log error messages informing you that the license has expired. These messages will also be generated at node startup, to ensure you fail to update it:
that there are no surprises. Here is an example message:
[source,sh] [source,sh]
--------------------------------------------------------------------------------------------------------------------------------- --------------------------------------------
[ERROR][shield.license] Shield license will expire on 1/1/1970. Cluster health, cluster stats and indices stats operations are # License will expire on [Thursday, November 12, 2015]. If you have a new license, please update it.
blocked on Shield license expiration. All data operations (read and write) continue to work. If you have a new license, please # Otherwise, please reach out to your support contact.
update it. Otherwise, please reach out to your support contact. #
--------------------------------------------------------------------------------------------------------------------------------- # Commercial plugins operate with reduced functionality on license expiration:
# - marvel
# - The agent will stop collecting cluster and indices metrics
# - shield
# - Cluster health, cluster stats and indices stats operations are blocked
# - All data operations (read and write) continue to work
--------------------------------------------
When the license for Shield is expired, Shield will block requests to the cluster health, cluster stats, and index stats APIs. Once the license expires, calls to the cluster health, cluster stats, and index stats APIs
Calls to these APIs will fail with a LicenseExpiredException, and will return HTTP status code 401. By disabling only these APIs, fail with a `ElasticsearchSecurityException` and return a 401 HTTP status code:
any automated cluster monitoring should detect the license failure, while users of the cluster should not be immediately impacted.
It is not recommended to run for any length of time with a disabled Shield license; cluster health and stats APIs are critical
for monitoring and management of an Elasticsearch cluster.
Example error response the clients will receive when license is expired and cluster health, cluster stats or index stats APIs are called: [source,sh]
--------------------------------------------
{
"error": {
"root_cause": [{
"type": "security_exception",
"reason": "current license is non-compliant for [shield]",
"license.expired.feature": "shield"
}],
"type": "security_exception",
"reason": "current license is non-compliant for [shield]",
"license.expired.feature": "shield"
},
"status": 401
}
--------------------------------------------
[source,json] This enables automatic monitoring systems to easily detect the license failure without immediately impacting other users.
----------------------------------------------------------------------------------------------------------------------------------------------
{"error":"LicenseExpiredException[license expired for feature [shield]]","status":401} IMPORTANT: You should update your license as soon as possible. You're essentially flying blind when running with an expired license. Access to the cluster health and stats APIs is critical
---------------------------------------------------------------------------------------------------------------------------------------------- for monitoring and managing an Elasticsearch cluster.
If you receive a new license file and <<installing-license, install it>>, it will take effect immediately and the health and
stats APIs will be available.