2018-10-25 13:32:50 -04:00
|
|
|
[role="xpack"]
|
|
|
|
[testenv="gold"]
|
|
|
|
[[configuring-metricbeat]]
|
2019-09-24 13:35:06 -04:00
|
|
|
== Collecting {es} monitoring data with {metricbeat}
|
2019-04-30 13:46:13 -04:00
|
|
|
|
|
|
|
[subs="attributes"]
|
2018-11-28 15:36:29 -05:00
|
|
|
++++
|
|
|
|
<titleabbrev>Collecting monitoring data with {metricbeat}</titleabbrev>
|
|
|
|
++++
|
2018-10-25 13:32:50 -04:00
|
|
|
|
2018-11-01 17:27:49 -04:00
|
|
|
In 6.5 and later, you can use {metricbeat} to collect data about {es}
|
2018-10-25 13:32:50 -04:00
|
|
|
and ship it to the monitoring cluster, rather than routing it through exporters
|
2018-11-28 15:36:29 -05:00
|
|
|
as described in <<collecting-monitoring-data>>.
|
2018-10-25 13:32:50 -04:00
|
|
|
|
|
|
|
image::monitoring/images/metricbeat.png[Example monitoring architecture]
|
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
//NOTE: The tagged regions are re-used in the Stack Overview.
|
|
|
|
|
2019-05-29 14:00:47 -04:00
|
|
|
. Enable the collection of monitoring data. +
|
|
|
|
+
|
2018-10-25 13:32:50 -04:00
|
|
|
--
|
2019-05-29 14:00:47 -04:00
|
|
|
// tag::enable-collection[]
|
2019-06-12 09:25:30 -04:00
|
|
|
Set `xpack.monitoring.collection.enabled` to `true` on the
|
2019-05-29 14:00:47 -04:00
|
|
|
production cluster. By default, it is is disabled (`false`).
|
|
|
|
|
2019-06-12 09:25:30 -04:00
|
|
|
You can use the following APIs to review and change this setting:
|
2018-10-25 13:32:50 -04:00
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2018-10-25 13:32:50 -04:00
|
|
|
----------------------------------
|
|
|
|
GET _cluster/settings
|
|
|
|
|
|
|
|
PUT _cluster/settings
|
|
|
|
{
|
|
|
|
"persistent": {
|
|
|
|
"xpack.monitoring.collection.enabled": true
|
|
|
|
}
|
|
|
|
}
|
2019-09-09 12:35:50 -04:00
|
|
|
----------------------------------
|
2019-05-29 14:00:47 -04:00
|
|
|
|
2019-06-12 09:25:30 -04:00
|
|
|
If {es} {security-features} are enabled, you must have `monitor` cluster privileges to
|
2018-11-08 16:53:09 -05:00
|
|
|
view the cluster settings and `manage` cluster privileges to change them.
|
|
|
|
|
2019-06-12 09:25:30 -04:00
|
|
|
// end::enable-collection[]
|
|
|
|
For more information, see <<monitoring-settings>> and <<cluster-update-settings>>.
|
2018-10-25 13:32:50 -04:00
|
|
|
--
|
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
. {metricbeat-ref}/metricbeat-installation.html[Install {metricbeat}] on each
|
|
|
|
{es} node in the production cluster.
|
2018-10-25 13:32:50 -04:00
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
. Enable the {es} {xpack} module in {metricbeat} on each {es} node. +
|
2018-10-25 13:32:50 -04:00
|
|
|
+
|
|
|
|
--
|
2019-05-28 12:04:02 -04:00
|
|
|
// tag::enable-es-module[]
|
2018-10-25 13:32:50 -04:00
|
|
|
For example, to enable the default configuration in the `modules.d` directory,
|
|
|
|
run the following command:
|
|
|
|
|
|
|
|
["source","sh",subs="attributes,callouts"]
|
|
|
|
----------------------------------------------------------------------
|
2019-04-08 12:31:26 -04:00
|
|
|
metricbeat modules enable elasticsearch-xpack
|
2018-10-25 13:32:50 -04:00
|
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
|
|
For more information, see
|
|
|
|
{metricbeat-ref}/configuration-metricbeat.html[Specify which modules to run] and
|
|
|
|
{metricbeat-ref}/metricbeat-module-elasticsearch.html[{es} module].
|
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
// end::enable-es-module[]
|
|
|
|
--
|
2018-10-25 13:32:50 -04:00
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
. Configure the {es} {xpack} module in {metricbeat} on each {es} node. +
|
2018-10-25 13:32:50 -04:00
|
|
|
+
|
|
|
|
--
|
2019-05-28 12:04:02 -04:00
|
|
|
// tag::configure-es-module[]
|
|
|
|
The `modules.d/elasticsearch-xpack.yml` file contains the following settings:
|
2018-10-25 13:32:50 -04:00
|
|
|
|
|
|
|
[source,yaml]
|
|
|
|
----------------------------------
|
2019-05-28 12:04:02 -04:00
|
|
|
- module: elasticsearch
|
|
|
|
metricsets:
|
|
|
|
- ccr
|
|
|
|
- cluster_stats
|
|
|
|
- index
|
|
|
|
- index_recovery
|
|
|
|
- index_summary
|
|
|
|
- ml_job
|
|
|
|
- node_stats
|
|
|
|
- shard
|
|
|
|
period: 10s
|
|
|
|
hosts: ["http://localhost:9200"]
|
|
|
|
#username: "user"
|
|
|
|
#password: "secret"
|
|
|
|
xpack.enabled: true
|
2018-10-25 13:32:50 -04:00
|
|
|
----------------------------------
|
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
By default, the module collects {es} monitoring metrics from
|
|
|
|
`http://localhost:9200`. If that host and port number are not correct, you must
|
|
|
|
update the `hosts` setting. If you configured {es} to use encrypted
|
|
|
|
communications, you must access it via HTTPS. For example, use a `hosts` setting
|
|
|
|
like `https://localhost:9200`.
|
|
|
|
// end::configure-es-module[]
|
2018-10-25 13:32:50 -04:00
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
// tag::remote-monitoring-user[]
|
|
|
|
If Elastic {security-features} are enabled, you must also provide a user ID
|
|
|
|
and password so that {metricbeat} can collect metrics successfully:
|
|
|
|
|
|
|
|
.. Create a user on the production cluster that has the
|
2019-10-07 18:23:19 -04:00
|
|
|
<<built-in-roles,`remote_monitoring_collector` built-in role>>.
|
2019-05-28 12:04:02 -04:00
|
|
|
Alternatively, use the
|
2019-10-07 18:23:19 -04:00
|
|
|
<<built-in-users,`remote_monitoring_user` built-in user>>.
|
2019-05-28 12:04:02 -04:00
|
|
|
|
|
|
|
.. Add the `username` and `password` settings to the {es} module configuration
|
|
|
|
file.
|
|
|
|
// end::remote-monitoring-user[]
|
|
|
|
--
|
|
|
|
|
2019-05-30 15:12:37 -04:00
|
|
|
. Optional: Disable the system module in {metricbeat}.
|
|
|
|
+
|
|
|
|
--
|
|
|
|
// tag::disable-system-module[]
|
|
|
|
By default, the {metricbeat-ref}/metricbeat-module-system.html[system module] is
|
|
|
|
enabled. The information it collects, however, is not shown on the *Monitoring*
|
|
|
|
page in {kib}. Unless you want to use that information for other purposes, run
|
|
|
|
the following command:
|
|
|
|
|
|
|
|
["source","sh",subs="attributes,callouts"]
|
|
|
|
----------------------------------------------------------------------
|
|
|
|
metricbeat modules disable system
|
|
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
|
|
// end::disable-system-module[]
|
|
|
|
--
|
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
. Identify where to send the monitoring data. +
|
2018-10-25 13:32:50 -04:00
|
|
|
+
|
|
|
|
--
|
|
|
|
TIP: In production environments, we strongly recommend using a separate cluster
|
|
|
|
(referred to as the _monitoring cluster_) to store the data. Using a separate
|
|
|
|
monitoring cluster prevents production cluster outages from impacting your
|
|
|
|
ability to access your monitoring data. It also prevents monitoring activities
|
|
|
|
from impacting the performance of your production cluster.
|
|
|
|
|
|
|
|
For example, specify the {es} output information in the {metricbeat}
|
|
|
|
configuration file (`metricbeat.yml`):
|
|
|
|
|
|
|
|
[source,yaml]
|
|
|
|
----------------------------------
|
|
|
|
output.elasticsearch:
|
2019-05-28 12:04:02 -04:00
|
|
|
# Array of hosts to connect to.
|
2018-10-25 13:32:50 -04:00
|
|
|
hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] <1>
|
2019-05-28 12:04:02 -04:00
|
|
|
|
|
|
|
# Optional protocol and basic auth credentials.
|
|
|
|
#protocol: "https"
|
|
|
|
#username: "elastic"
|
|
|
|
#password: "changeme"
|
2018-10-25 13:32:50 -04:00
|
|
|
----------------------------------
|
|
|
|
<1> In this example, the data is stored on a monitoring cluster with nodes
|
|
|
|
`es-mon-1` and `es-mon-2`.
|
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
If you configured the monitoring cluster to use encrypted communications, you
|
|
|
|
must access it via HTTPS. For example, use a `hosts` setting like
|
|
|
|
`https://es-mon-1:9200`.
|
|
|
|
|
2018-12-17 11:18:11 -05:00
|
|
|
IMPORTANT: The {es} {monitor-features} use ingest pipelines, therefore the
|
|
|
|
cluster that stores the monitoring data must have at least one
|
|
|
|
<<ingest,ingest node>>.
|
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
If {es} {security-features} are enabled on the monitoring cluster, you must
|
|
|
|
provide a valid user ID and password so that {metricbeat} can send metrics
|
|
|
|
successfully:
|
2018-10-25 13:32:50 -04:00
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
.. Create a user on the monitoring cluster that has the
|
2019-10-07 18:23:19 -04:00
|
|
|
<<built-in-roles,`remote_monitoring_agent` built-in role>>.
|
2018-11-08 16:53:09 -05:00
|
|
|
Alternatively, use the
|
2019-10-07 18:23:19 -04:00
|
|
|
<<built-in-users,`remote_monitoring_user` built-in user>>.
|
2018-10-25 13:32:50 -04:00
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
.. Add the `username` and `password` settings to the {es} output information in
|
|
|
|
the {metricbeat} configuration file.
|
2018-10-25 13:32:50 -04:00
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
For more information about these configuration options, see
|
|
|
|
{metricbeat-ref}/elasticsearch-output.html[Configure the {es} output].
|
|
|
|
--
|
2018-10-25 13:32:50 -04:00
|
|
|
|
2019-05-28 12:04:02 -04:00
|
|
|
. {metricbeat-ref}/metricbeat-starting.html[Start {metricbeat}] on each node.
|
2018-10-25 13:32:50 -04:00
|
|
|
|
2019-06-12 09:25:30 -04:00
|
|
|
. Disable the default collection of {es} monitoring metrics. +
|
|
|
|
+
|
|
|
|
--
|
|
|
|
// tag::disable-default-collection[]
|
|
|
|
Set `xpack.monitoring.elasticsearch.collection.enabled` to `false` on the
|
|
|
|
production cluster.
|
|
|
|
|
|
|
|
You can use the following API to change this setting:
|
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2019-06-12 09:25:30 -04:00
|
|
|
----------------------------------
|
|
|
|
PUT _cluster/settings
|
|
|
|
{
|
|
|
|
"persistent": {
|
|
|
|
"xpack.monitoring.elasticsearch.collection.enabled": false
|
|
|
|
}
|
|
|
|
}
|
|
|
|
----------------------------------
|
|
|
|
|
|
|
|
If {es} {security-features} are enabled, you must have `monitor` cluster
|
|
|
|
privileges to view the cluster settings and `manage` cluster privileges
|
|
|
|
to change them.
|
|
|
|
|
|
|
|
// end::disable-default-collection[]
|
|
|
|
--
|
|
|
|
|
2018-10-25 13:32:50 -04:00
|
|
|
. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}].
|