[role="xpack"] [testenv="gold"] [[collecting-monitoring-data]] == Collecting monitoring data [IMPORTANT] ========================= {metricbeat} is the recommended method for collecting and shipping monitoring data to a monitoring cluster. If you have previously configured internal collection, you should migrate to using {metricbeat} collection. Use either {metricbeat} collection or internal collection; do not use both. Learn more about <>. ========================= If you enable the Elastic {monitor-features} in your cluster, you can optionally collect metrics about {es}. By default, monitoring is enabled but data collection is disabled. This method involves sending the metrics to the monitoring cluster by using exporters. For the recommended method, see <>. Advanced monitoring settings enable you to control how frequently data is collected, configure timeouts, and set the retention period for locally-stored monitoring indices. You can also adjust how monitoring data is displayed. To learn about monitoring in general, see <>. . Configure your cluster to collect monitoring data: .. Verify that the `xpack.monitoring.enabled` setting is `true`, which is its default value, on each node in the cluster. For more information, see <>. .. Verify that the `xpack.monitoring.elasticsearch.collection.enabled` setting is `true`, which is its default value, on each node in the cluster. + -- NOTE: You can specify this setting in either the `elasticsearch.yml` on each node or across the cluster as a dynamic cluster setting. If {es} {security-features} are enabled, you must have `monitor` cluster privileges to view the cluster settings and `manage` cluster privileges to change them. For more information, see <> and <>. -- .. Set the `xpack.monitoring.collection.enabled` setting to `true` on each node in the cluster. By default, it is is disabled (`false`). + -- NOTE: You can specify this setting in either the `elasticsearch.yml` on each node or across the cluster as a dynamic cluster setting. If {es} {security-features} are enabled, you must have `monitor` cluster privileges to view the cluster settings and `manage` cluster privileges to change them. For example, use the following APIs to review and change this setting: [source,console] ---------------------------------- GET _cluster/settings PUT _cluster/settings { "persistent": { "xpack.monitoring.collection.enabled": true } } ---------------------------------- Alternatively, you can enable this setting in {kib}. In the side navigation, click *Monitoring*. If data collection is disabled, you are prompted to turn it on. For more information, see <> and <>. -- .. Optional: Specify which indices you want to monitor. + -- By default, the monitoring agent collects data from all {es} indices. To collect data from particular indices, configure the `xpack.monitoring.collection.indices` setting. You can specify multiple indices as a comma-separated list or use an index pattern to match multiple indices. For example: [source,yaml] ---------------------------------- xpack.monitoring.collection.indices: logstash-*, index1, test2 ---------------------------------- You can prepend `-` to explicitly exclude index names or patterns. For example, to include all indices that start with `test` except `test3`, you could specify `test*,-test3`. To include system indices such as .security and .kibana, add `.*` to the list of included names. For example `.*,test*,-test3` -- .. Optional: Specify how often to collect monitoring data. The default value for the `xpack.monitoring.collection.interval` setting 10 seconds. See <>. . Identify where to store monitoring data. + -- By default, the data is stored on the same cluster by using a <>. Alternatively, you can use an <> to send data to a separate _monitoring cluster_. IMPORTANT: The {es} {monitor-features} use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one <>. For more information about typical monitoring architectures, see <>. -- . If you choose to use an `http` exporter: .. On the cluster that you want to monitor (often called the _production cluster_), configure each node to send metrics to your monitoring cluster. Configure an HTTP exporter in the `xpack.monitoring.exporters` settings in the `elasticsearch.yml` file. For example: + -- [source,yaml] -------------------------------------------------- xpack.monitoring.exporters: id1: type: http host: ["http://es-mon-1:9200", "http://es-mon2:9200"] -------------------------------------------------- -- .. If the Elastic {security-features} are enabled on the monitoring cluster, you must provide appropriate credentials when data is shipped to the monitoring cluster: ... Create a user on the monitoring cluster that has the <>. Alternatively, use the <>. ... Add the user ID and password settings to the HTTP exporter settings in the `elasticsearch.yml` file on each node. + + -- For example: [source,yaml] -------------------------------------------------- xpack.monitoring.exporters: id1: type: http host: ["http://es-mon-1:9200", "http://es-mon2:9200"] auth.username: remote_monitoring_user auth.password: YOUR_PASSWORD -------------------------------------------------- -- .. If you configured the monitoring cluster to use <>, you must use the HTTPS protocol in the `host` setting. You must also specify the trusted CA certificates that will be used to verify the identity of the nodes in the monitoring cluster. *** To add a CA certificate to an {es} node's trusted certificates, you can specify the location of the PEM encoded certificate with the `certificate_authorities` setting. For example: + -- [source,yaml] -------------------------------------------------- xpack.monitoring.exporters: id1: type: http host: ["https://es-mon1:9200", "https://es-mon2:9200"] auth: username: remote_monitoring_user password: YOUR_PASSWORD ssl: certificate_authorities: [ "/path/to/ca.crt" ] -------------------------------------------------- -- *** Alternatively, you can configure trusted certificates using a truststore (a Java Keystore file that contains the certificates). For example: + -- [source,yaml] -------------------------------------------------- xpack.monitoring.exporters: id1: type: http host: ["https://es-mon1:9200", "https://es-mon2:9200"] auth: username: remote_monitoring_user password: YOUR_PASSWORD ssl: truststore.path: /path/to/file truststore.password: password -------------------------------------------------- -- . Configure your cluster to route monitoring data from sources such as {kib}, Beats, and {ls} to the monitoring cluster. For information about configuring each product to collect and send monitoring data, see <>. . If you updated settings in the `elasticsearch.yml` files on your production cluster, restart {es}. See <> and <>. + -- TIP: You may want to temporarily {ref}/modules-cluster.html[disable shard allocation] before you restart your nodes to avoid unnecessary shard reallocation during the install process. -- . Optional: <>. . {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}].