OpenSearch/docs/reference/monitoring/production.asciidoc

[role="xpack"]
[[monitoring-production]]
== Monitoring in a production environment

In production, you should send monitoring data to a separate _monitoring cluster_
so that historical data is available even when the nodes you are monitoring are
not. For example, you can use {metricbeat} to ship monitoring data about {kib},
{es}, {ls}, and Beats to the monitoring cluster.

[IMPORTANT]
=========================
{metricbeat} is the recommended method for collecting and shipping monitoring
data to a monitoring cluster.

If you have previously configured legacy collection methods, you should migrate
to using {metricbeat} collection. Use either {metricbeat} collection or
legacy collection methods; do not use both.

Learn more about <<configuring-metricbeat>>.
=========================

If you have at least a Gold Subscription, using a dedicated monitoring cluster
also enables you to monitor multiple clusters from a central location.

To store monitoring data in a separate cluster:

. Set up the {es} cluster you want to use as the monitoring cluster.
For example, you might set up a two host cluster with the nodes `es-mon-1` and
`es-mon-2`.
+
--
[IMPORTANT]
===============================
* Ideally the monitoring cluster and the production cluster run on the same
{stack} version. However, a monitoring cluster on the latest release of
{major-version} also works with production clusters that use the same major
version. Monitoring clusters that use {major-version} also work with production
clusters that use the latest release of {prev-major-version}.
* There must be at least one <<ingest,ingest node>> in the monitoring
cluster; it does not need to be a dedicated ingest node.
===============================
--

.. (Optional) Verify that the collection of monitoring data is disabled on the
monitoring cluster. By default, the `xpack.monitoring.collection.enabled` setting
is `false`.
+
--
For example, you can use the following APIs to review and change this setting:

[source,console]
----------------------------------
GET _cluster/settings

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": false
  }
}
----------------------------------
// TEST[skip:security errs]
--

.. If the {es} {security-features} are enabled on the monitoring cluster, create
users that can send and retrieve monitoring data.
+
--
NOTE: If you plan to use {kib} to view monitoring data, username and password
credentials must be valid on both the {kib} server and the monitoring cluster.

--

*** If you plan to use {metricbeat} to collect data about {es} or {kib},
create a user that has the `remote_monitoring_collector` built-in role and a
user that has the `remote_monitoring_agent`
<<built-in-roles-remote-monitoring-agent,built-in role>>. Alternatively, use the
`remote_monitoring_user` <<built-in-users,built-in user>>.

*** If you plan to use HTTP exporters to route data through your production
cluster, create a user that has the `remote_monitoring_agent`
<<built-in-roles-remote-monitoring-agent,built-in role>>.
+
--
For example, the
following request creates a `remote_monitor` user that has the
`remote_monitoring_agent` role:

[source,console]
---------------------------------------------------------------
POST /_security/user/remote_monitor
{
  "password" : "changeme",
  "roles" : [ "remote_monitoring_agent"],
  "full_name" : "Internal Agent For Remote Monitoring"
}
---------------------------------------------------------------
// TEST[skip:needs-gold+-license]

Alternatively, use the `remote_monitoring_user` <<built-in-users,built-in user>>.
--

. Configure your production cluster to collect data and send it to the
monitoring cluster:

** <<configuring-metricbeat,{metricbeat} collection methods>>

** <<collecting-monitoring-data,Legacy collection methods>>

. (Optional)
{logstash-ref}/configuring-logstash.html[Configure {ls} to collect data and send it to the monitoring cluster].

. (Optional) Configure the Beats to collect data and send it to the monitoring
cluster.
** {auditbeat-ref}/monitoring.html[Auditbeat]
** {filebeat-ref}/monitoring.html[Filebeat]
** {heartbeat-ref}/monitoring.html[Heartbeat]
** {metricbeat-ref}/monitoring.html[Metricbeat]
** {packetbeat-ref}/monitoring.html[Packetbeat]
** {winlogbeat-ref}/monitoring.html[Winlogbeat]

. (Optional) Configure {kib} to collect data and send it to the monitoring cluster:

** {kibana-ref}/monitoring-metricbeat.html[{metricbeat} collection methods]

** {kibana-ref}/monitoring-kibana.html[Legacy collection methods]

. (Optional) Create a dedicated {kib} instance for monitoring, rather than using
a single {kib} instance to access both your production cluster and monitoring
cluster.
+
--
NOTE: If you log in to {kib} using SAML, Kerberos, PKI, OpenID Connect, or token
authentication providers, a dedicated {kib} instance is *required*. The security
tokens that are used in these contexts are cluster-specific, therefore you
cannot use a single {kib} instance to connect to both production and monitoring
clusters.

--

.. (Optional) Disable the collection of monitoring data in this {kib} instance.
Set the `xpack.monitoring.kibana.collection.enabled` setting to `false` in the
`kibana.yml` file. For more information about this setting, see
{kibana-ref}/monitoring-settings-kb.html[Monitoring settings in {kib}].

. {kibana-ref}/monitoring-data.html[Configure {kib} to retrieve and display the monitoring data].