diff --git a/docs/reference/monitoring/configuring-metricbeat.asciidoc b/docs/reference/monitoring/configuring-metricbeat.asciidoc index 34cbee9c169..d7bd58eb33f 100644 --- a/docs/reference/monitoring/configuring-metricbeat.asciidoc +++ b/docs/reference/monitoring/configuring-metricbeat.asciidoc @@ -15,9 +15,15 @@ To learn about monitoring in general, see {stack-ov}/xpack-monitoring.html[Monitoring the {stack}]. . Enable the collection of monitoring data. Set -`xpack.monitoring.collection.enabled` to `true` on the production cluster. + +`xpack.monitoring.collection.enabled` to `true` on each node in the production +cluster. By default, it is is disabled (`false`). + -- +NOTE: You can specify this setting in either the `elasticsearch.yml` on each +node or across the cluster as a dynamic cluster setting. If {es} +{security-features} are enabled, you must have `monitor` cluster privileges to +view the cluster settings and `manage` cluster privileges to change them. + For example, you can use the following APIs to review and change this setting: [source,js] @@ -31,16 +37,21 @@ PUT _cluster/settings } } ---------------------------------- -// CONSOLE +// CONSOLE For more information, see <> and <>. -- . Disable the default collection of {es} monitoring metrics. Set -`xpack.monitoring.elasticsearch.collection.enabled` to `false` on the production -cluster. + +`xpack.monitoring.elasticsearch.collection.enabled` to `false` on each node in +the production cluster. + -- +NOTE: You can specify this setting in either the `elasticsearch.yml` on each +node or across the cluster as a dynamic cluster setting. If {es} +{security-features} are enabled, you must have `monitor` cluster privileges to +view the cluster settings and `manage` cluster privileges to change them. + For example, you can use the following API to change this setting: [source,js] @@ -96,30 +107,26 @@ You must specify the following settings in the `modules.d/elasticsearch.yml` fil - shard period: 10s hosts: ["http://localhost:9200"] <1> - xpack.enabled: true + xpack.enabled: true <2> ---------------------------------- -<1> This setting identifies the host and port number that are used to access {es}. +<1> This setting identifies the host and port number that are used to access {es}. +<2> This setting ensures that {kib} can read this monitoring data successfully. +That is to say, it's stored in the same location and format as monitoring data +that is sent by <>. -- -.. If {security} is enabled, you must also provide a user ID and password so that -{metricbeat} can collect metrics successfully. +.. If Elastic {security-features} are enabled, you must also provide a user ID +and password so that {metricbeat} can collect metrics successfully. -... Create or identify a user that you want to use to collect the metrics. -+ --- -TIP: There is a `remote_monitoring_user` built-in user that grants the privileges -necessary for {metricbeat} to monitor {stack} products. See -{stack-ov}/built-in-users.html[Built-in users]. - -Alternatively, you can choose a different user and give them the -`remote_monitoring_collector` {stack-ov}/built-in-roles.html[built-in role]. --- +... Create a user on the production cluster that has the +{stack-ov}/built-in-roles.html[`remote_monitoring_collector` built-in role]. +Alternatively, use the {stack-ov}/built-in-users.html[`remote_monitoring_user` built-in user]. ... Add the `username` and `password` settings to the {es} module configuration file. + -- -For example, add the following settings in the `modules.d/kibana.yml` file: +For example, add the following settings in the `modules.d/elasticsearch.yml` file: [source,yaml] ---------------------------------- @@ -158,19 +165,14 @@ For more information about these configuration options, see {metricbeat-ref}/elasticsearch-output.html[Configure the {es} output]. -- -.. If {security} is enabled on the monitoring cluster, you must provide a valid -user ID and password so that {metricbeat} can send metrics successfully. +.. If {es} {security-features} are enabled on the monitoring cluster, you +must provide a valid user ID and password so that {metricbeat} can send metrics +successfully. -... Create or identify a user that you want to use to send the metrics. -+ --- -TIP: There is a `remote_monitoring_user` built-in user that grants the privileges -necessary for {metricbeat} to monitor {stack} products. See -{stack-ov}/built-in-users.html[Built-in users]. - -Alternatively, you can choose a different user and give them the -`remote_monitoring_agent` {stack-ov}/built-in-roles.html[built-in role]. --- +... Create a user on the monitoring cluster that has the +{stack-ov}/built-in-roles.html[`remote_monitoring_agent` built-in role]. +Alternatively, use the +{stack-ov}/built-in-users.html[`remote_monitoring_user` built-in user]. ... Add the `username` and `password` settings to the {es} output information in the {metricbeat} configuration file (`metricbeat.yml`): diff --git a/docs/reference/monitoring/configuring-monitoring.asciidoc b/docs/reference/monitoring/configuring-monitoring.asciidoc index 81a9cce4f12..e7ed3a45397 100644 --- a/docs/reference/monitoring/configuring-monitoring.asciidoc +++ b/docs/reference/monitoring/configuring-monitoring.asciidoc @@ -6,8 +6,8 @@ Configuring monitoring ++++ -If you enable the collection of monitoring data in your cluster, you can -optionally collect metrics about {es}. By default, {monitoring} is enabled but +If you enable the Elastic {monitor-features} in your cluster, you can +optionally collect metrics about {es}. By default, monitoring is enabled but data collection is disabled. The following method involves sending the metrics to the monitoring cluster by @@ -20,15 +20,51 @@ monitoring indices. You can also adjust how monitoring data is displayed. To learn about monitoring in general, see {stack-ov}/xpack-monitoring.html[Monitoring the {stack}]. -. To collect monitoring data about your {es} cluster: +. Configure your cluster to collect monitoring data: -.. Verify that the `xpack.monitoring.enabled`, -`xpack.monitoring.collection.enabled`, and -`xpack.monitoring.elasticsearch.collection.enabled` settings are `true` on each -node in the cluster. By default `xpack.monitoring.collection.enabled` is disabled -(`false`), and that overrides `xpack.monitoring.elasticsearch.collection.enabled`, -which defaults to being enabled (`true`). Both settings can be set dynamically -at runtime. For more information, see <>. +.. Verify that the `xpack.monitoring.enabled` setting is `true`, which is its +default value, on each node in the cluster. For more information, see +<>. + +.. Verify that the `xpack.monitoring.elasticsearch.collection.enabled` setting +is `true`, which is its default value, on each node in the cluster. ++ +-- +NOTE: You can specify this setting in either the `elasticsearch.yml` on each +node or across the cluster as a dynamic cluster setting. If {es} +{security-features} are enabled, you must have `monitor` cluster privileges to +view the cluster settings and `manage` cluster privileges to change them. + +For more information, see <> and <>. +-- + +.. Set the `xpack.monitoring.collection.enabled` setting to `true` on each +node in the cluster. By default, it is is disabled (`false`). ++ +-- +NOTE: You can specify this setting in either the `elasticsearch.yml` on each +node or across the cluster as a dynamic cluster setting. If {es} +{security-features} are enabled, you must have `monitor` cluster privileges to +view the cluster settings and `manage` cluster privileges to change them. + +For example, use the following APIs to review and change this setting: + +[source,js] +---------------------------------- +GET _cluster/settings + +PUT _cluster/settings +{ + "persistent": { + "xpack.monitoring.collection.enabled": true + } +} +---------------------------------- +// CONSOLE + +For more +information, see <> and <>. +-- .. Optional: Specify which indices you want to monitor. + @@ -53,62 +89,71 @@ patterns. For example, to include all indices that start with `test` except the `xpack.monitoring.collection.interval` setting 10 seconds. See <>. -. Optional: Configure your cluster to route monitoring data from sources such -as {kib}, Beats, and Logstash to a monitoring cluster: - -.. Verify that `xpack.monitoring.collection.enabled` settings are `true` on each -node in the cluster. - -.. {stack-ov}/xpack-monitoring.html[Configure {monitoring} across the Elastic Stack]. - . Identify where to store monitoring data. + -- -By default, {monitoring} uses a `local` exporter that indexes monitoring data -on the same cluster. See <> and <>. +By default, the data is stored on the same cluster by using a +<>. -Alternatively, you can use an `http` exporter to send data to a separate -monitoring cluster. See <>. +Alternatively, you can use an <> to send data to +a separate _monitoring cluster_. For more information about typical monitoring architectures, see {stack-ov}/how-monitoring-works.html[How Monitoring Works]. -- -. If {security} is enabled and you are using an `http` exporter to send data to - a dedicated monitoring cluster: +. If you choose to use an `http` exporter: -.. Create a user on the monitoring cluster that has the -{stack-ov}/built-in-roles.html#built-in-roles-remote-monitoring-agent[`remote_monitoring_agent` built-in role]. -For example, the following request creates a `remote_monitor` user that has the -`remote_monitoring_agent` role: +.. On the cluster that you want to monitor (often called the _production cluster_), +configure each node to send metrics to your monitoring cluster. Configure an +HTTP exporter in the `xpack.monitoring.exporters` settings in the +`elasticsearch.yml` file. For example: + -- -[source, sh] ---------------------------------------------------------------- -POST /_xpack/security/user/remote_monitor -{ - "password" : "changeme", - "roles" : [ "remote_monitoring_agent"], - "full_name" : "Internal Agent For Remote Monitoring" -} ---------------------------------------------------------------- -// CONSOLE -// TEST[skip:needs-gold+-license] +[source,yaml] +-------------------------------------------------- +xpack.monitoring.exporters: + id1: + type: http + host: ["http://es-mon-1:9200", "http://es-mon2:9200"] +-------------------------------------------------- -- -.. On each node in the cluster that is being monitored, configure the `http` -exporter to use the appropriate credentials when data is shipped to the -monitoring cluster. +.. If the Elastic {security-features} are enabled on the monitoring cluster, you +must provide appropriate credentials when data is shipped to the monitoring cluster: + +... Create a user on the monitoring cluster that has the +{stack-ov}/built-in-roles.html[`remote_monitoring_agent` built-in role]. +Alternatively, use the +{stack-ov}/built-in-users.html[`remote_monitoring_user` built-in user]. + +... Add the user ID and password settings to the HTTP exporter settings in the +`elasticsearch.yml` file on each node. + + -- -If SSL/TLS is enabled on the monitoring cluster, you must use the HTTPS protocol -in the `host` setting. You must also include the CA certificate in each node's -trusted certificates in order to verify the identities of the nodes in the -monitoring cluster. +For example: -The following example specifies the location of the PEM encoded certificate with -the `certificate_authorities` setting: +[source,yaml] +-------------------------------------------------- +xpack.monitoring.exporters: + id1: + type: http + host: ["http://es-mon-1:9200", "http://es-mon2:9200"] + auth.username: remote_monitoring_user + auth.password: YOUR_PASSWORD +-------------------------------------------------- +-- +.. If you configured the monitoring cluster to use +<>, you must use the HTTPS protocol in +the `host` setting. You must also specify the trusted CA certificates that will +be used to verify the identity of the nodes in the monitoring cluster. + +*** To add a CA certificate to an {es} node's trusted certificates, you can +specify the location of the PEM encoded certificate with the +`certificate_authorities` setting. For example: ++ +-- [source,yaml] -------------------------------------------------- xpack.monitoring.exporters: @@ -116,18 +161,17 @@ xpack.monitoring.exporters: type: http host: ["https://es-mon1:9200", "https://es-mon2:9200"] auth: - username: remote_monitor <1> - password: changeme + username: remote_monitoring_user + password: YOUR_PASSWORD ssl: certificate_authorities: [ "/path/to/ca.crt" ] - id2: - type: local -------------------------------------------------- -<1> The `username` and `password` parameters provide the user credentials. - -Alternatively, you can configure trusted certificates using a truststore -(a Java Keystore file that contains the certificates): +-- +*** Alternatively, you can configure trusted certificates using a truststore +(a Java Keystore file that contains the certificates). For example: ++ +-- [source,yaml] -------------------------------------------------- xpack.monitoring.exporters: @@ -135,25 +179,28 @@ xpack.monitoring.exporters: type: http host: ["https://es-mon1:9200", "https://es-mon2:9200"] auth: - username: remote_monitor - password: changeme + username: remote_monitoring_user + password: YOUR_PASSWORD ssl: truststore.path: /path/to/file truststore.password: password - id2: - type: local -------------------------------------------------- -- -. If {security} is enabled and you want to visualize monitoring data in {kib}, -you must create users that have access to the {kib} indices and permission to -read from the monitoring indices. +. Configure your cluster to route monitoring data from sources such as {kib}, +Beats, and {ls} to the monitoring cluster. The +`xpack.monitoring.collection.enabled` setting must be `true` on each node in the +cluster. For information about configuring each product to collect and send +monitoring data, see {stack-ov}/xpack-monitoring.html[Monitoring the {stack}]. + +. If you updated settings in the `elasticsearch.yml` files on your production +cluster, restart {es}. See <> and <>. + -- -You set up {monitoring} UI users on the cluster where the monitoring data is -stored, that is to say the monitoring cluster. To grant all of the necessary permissions, assign users the -`monitoring_user` and `kibana_user` roles. For more information, see -{stack-ov}/mapping-roles.html[Mapping users and groups to roles]. +TIP: You may want to temporarily {ref}/modules-cluster.html[disable shard +allocation] before you restart your nodes to avoid unnecessary shard +reallocation during the install process. + -- . Optional: diff --git a/docs/reference/monitoring/index.asciidoc b/docs/reference/monitoring/index.asciidoc index d6a55f44585..13e7314f8af 100644 --- a/docs/reference/monitoring/index.asciidoc +++ b/docs/reference/monitoring/index.asciidoc @@ -5,38 +5,46 @@ [partintro] -- -{monitoring} enables you to easily monitor the health of your {es} cluster. The -monitoring metrics are collected from each node and stored in {es} indices. +The Elastic {monitor-features} enable you to easily monitor the health of +your {es} cluster. The monitoring metrics are collected from each node and +stored in {es} indices. + +TIP: In production environments, it is recommended to store the monitoring data +in a separate _monitoring cluster_. See +{stack-ov}/monitoring-production.html[Monitoring in a production environment]. Each {es} node is considered unique based on its persistent UUID, which is written on first start to its <> directory, which defaults to `./data`. -All settings associated with {monitoring} in {es} must be set in either the +All settings associated with monitoring in {es} must be set in either the `elasticsearch.yml` file for each node or, where possible, in the dynamic cluster settings. For more information, see <>. [[es-monitoring-overview]] -{es} is also at the core of {monitoring} across the Elastic Stack. In all cases, -{monitoring} documents are just ordinary JSON documents built by monitoring each -Elastic Stack component at some collection interval, then indexing those -documents into the monitoring cluster. Each component in the stack is -responsible for monitoring itself and then forwarding those documents to {es} -for both routing and indexing (storage). +{es} is also at the core of monitoring across the {stack}. In all cases, +monitoring documents are just ordinary JSON documents built by monitoring each +{stack} component at some collection interval, then indexing those +documents into the monitoring cluster. -The routing and indexing processes in {es} are handled by what are called -<> and -<>. In the past, collectors and exporters -were considered to be part of a monitoring "agent", but that term is generally -not used anymore. +Each component in the stack is responsible for monitoring itself and then +forwarding those documents to the {es} production cluster for both routing and +indexing (storage). The routing and indexing processes in {es} are handled by +what are called <> and +<>. + +beta[] Alternatively, in 6.4 and later, you can use {metricbeat} to collect +monitoring data about {kib} and ship it directly to the monitoring cluster, +rather than routing it through the production cluster. In 6.5 and later, you +can also use {metricbeat} to collect and ship data about {es}. You can view monitoring data from {kib} where it’s easy to spot issues at a glance or delve into the system behavior over time to diagnose operational issues. In addition to the built-in status warnings, you can also set up custom alerts based on the data in the monitoring indices. -For an introduction to monitoring your Elastic stack, including Beats, Logstash, -and {kib}, see {xpack-ref}/xpack-monitoring.html[Monitoring the Elastic Stack]. +For an introduction to monitoring your {stack}, including Beats, {ls}, and {kib}, +see {stack-ov}/xpack-monitoring.html[Monitoring the {stack}]. --