[DOCS] Adds production monitoring page (#47184)

docs/reference/monitoring/collecting-monitoring-data.asciidoc

@@ -107,7 +107,7 @@ cluster that stores the monitoring data must have at least one
 <<ingest,ingest node>>. 
 
 For more information about typical monitoring architectures, 
-see {stack-ov}/how-monitoring-works.html[How Monitoring Works].
+see <<how-monitoring-works>>.
 --
 
 . If you choose to use an `http` exporter: 

docs/reference/monitoring/collectors.asciidoc

@@ -140,4 +140,4 @@ related to monitoring data, which can be very useful when there are a large
 number of Logstash nodes or Beats.
 
 For more information about typical monitoring architectures, see
-{xpack-ref}/how-monitoring-works.html[How Monitoring Works].
+<<how-monitoring-works>>.

docs/reference/monitoring/configuring-filebeat.asciidoc

@@ -23,7 +23,7 @@ TIP: In production environments, we strongly recommend using a separate cluster
 monitoring cluster prevents production cluster outages from impacting your 
 ability to access your monitoring data. It also prevents monitoring activities 
 from impacting the performance of your production cluster. See
-{stack-ov}/monitoring-production.html[Monitoring in a production environment].
+<<monitoring-production>>.
 
 --
 

docs/reference/monitoring/index.asciidoc

@@ -10,12 +10,14 @@ performance of your {es} cluster.
 
 * <<monitoring-overview>>
 * <<how-monitoring-works>>
+* <<monitoring-production>>
 * <<collecting-monitoring-data>>
 * <<configuring-metricbeat>>
 * <<configuring-filebeat>>
 * <<config-monitoring-indices>>
 * <<es-monitoring-collectors>>
 * <<es-monitoring-exporters>>
+* <<monitoring-troubleshooting>>
 
 --
 
@@ -23,6 +25,8 @@ include::overview.asciidoc[]
 
 include::how-monitoring-works.asciidoc[]
 
+include::production.asciidoc[]
+
 include::collecting-monitoring-data.asciidoc[]
 include::pause-export.asciidoc[]
 
@@ -37,3 +41,5 @@ include::collectors.asciidoc[]
 include::exporters.asciidoc[]
 include::local-export.asciidoc[]
 include::http-export.asciidoc[]
+
+include::troubleshooting.asciidoc[]

docs/reference/monitoring/production.asciidoc

@@ -0,0 +1,124 @@
+[role="xpack"]
+[[monitoring-production]]
+== Monitoring in a production environment
+
+In production, you should send monitoring data to a separate _monitoring cluster_
+so that historical data is available even when the nodes you are monitoring are
+not. For example, you can use {metricbeat} to ship monitoring data about {kib},
+{es}, {ls}, and Beats to the monitoring cluster.
+//If you are sending your data to the {esms-init}, see <<esms>>.
+
+If you have at least a gold license, using a dedicated monitoring cluster also 
+enables you to monitor multiple clusters from a central location.
+
+To store monitoring data in a separate cluster:
+
+. Set up the {es} cluster you want to use as the monitoring cluster. 
+For example, you might set up a two host cluster with the nodes `es-mon-1` and 
+`es-mon-2`.
++
+--
+[IMPORTANT]
+===============================
+* To monitor an {es} {major-version} cluster, you must run {es} {major-version}
+on the monitoring cluster.
+* There must be at least one <<ingest,ingest node>> in the monitoring
+cluster; it does not need to be a dedicated ingest node.
+===============================
+--
+
+.. (Optional) Verify that the collection of monitoring data is disabled on the 
+monitoring cluster. By default, the `xpack.monitoring.collection.enabled` setting 
+is `false`. 
++ 
+--
+For example, you can use the following APIs to review and change this setting:
+
+[source,console]
+----------------------------------
+GET _cluster/settings
+
+PUT _cluster/settings
+{
+  "persistent": {
+    "xpack.monitoring.collection.enabled": false
+  }
+}
+----------------------------------
+
+--
+
+.. If the {es} {security-features} are enabled on the monitoring cluster, create 
+users that can send and retrieve monitoring data. 
++
+--
+NOTE: If you plan to use {kib} to view monitoring data, username and password 
+credentials must be valid on both the {kib} server and the monitoring cluster. 
+
+--
+
+*** If you plan to use {metricbeat} to collect data about {es} or {kib}, 
+create a user that has the `remote_monitoring_collector` built-in role and a 
+user that has the `remote_monitoring_agent` 
+{stack-ov}/built-in-roles.html#built-in-roles-remote-monitoring-agent[built-in role]. Alternatively, use the 
+`remote_monitoring_user` {stack-ov}/built-in-users.html[built-in user]. 
+
+*** If you plan to use HTTP exporters to route data through your production 
+cluster, create a user that has the `remote_monitoring_agent` 
+{stack-ov}/built-in-roles.html#built-in-roles-remote-monitoring-agent[built-in role]. 
++
+--
+For example, the 
+following request creates a `remote_monitor` user that has the 
+`remote_monitoring_agent` role:
+
+[source, sh]
+---------------------------------------------------------------
+POST /_security/user/remote_monitor
+{
+  "password" : "changeme",
+  "roles" : [ "remote_monitoring_agent"],
+  "full_name" : "Internal Agent For Remote Monitoring"
+}
+---------------------------------------------------------------
+// CONSOLE
+// TEST[skip:needs-gold+-license]
+
+Alternatively, use the `remote_monitoring_user` {stack-ov}/built-in-users.html[built-in user]. 
+--
+
+. Configure your production cluster to collect data and send it to the 
+monitoring cluster. 
+
+** <<configuring-metricbeat,Use {metricbeat}>>.
+
+** <<configuring-monitoring,Use HTTP exporters>>.
+
+. (Optional)
+{logstash-ref}/configuring-logstash.html[Configure {ls} to collect data and send it to the monitoring cluster].
+
+. (Optional) Configure the Beats to collect data and send it to the monitoring
+cluster. 
+** {auditbeat-ref}/monitoring.html[Auditbeat]
+** {filebeat-ref}/monitoring.html[Filebeat]
+** {heartbeat-ref}/monitoring.html[Heartbeat]
+** {metricbeat-ref}/monitoring.html[Metricbeat]
+** {packetbeat-ref}/monitoring.html[Packetbeat]
+** {winlogbeat-ref}/monitoring.html[Winlogbeat] 
+
+. (Optional) Configure {kib} to collect data and send it to the monitoring cluster:
+
+** {kibana-ref}/monitoring-metricbeat.html[Use {metricbeat}].
+
+** {kibana-ref}/monitoring-kibana.html[Use HTTP exporters].
+
+. (Optional) Create a dedicated {kib} instance for monitoring, rather than using 
+a single {kib} instance to access both your production cluster and monitoring 
+cluster.
+
+.. (Optional) Disable the collection of monitoring data in this {kib} instance. 
+Set the `xpack.monitoring.kibana.collection.enabled` setting to `false` in the 
+`kibana.yml` file. For more information about this setting, see 
+{kibana-ref}/monitoring-settings-kb.html[Monitoring settings in {kib}]. 
+
+. {kibana-ref}/monitoring-data.html[Configure {kib} to retrieve and display the monitoring data]. 

docs/reference/monitoring/troubleshooting.asciidoc

@@ -0,0 +1,29 @@
+[[monitoring-troubleshooting]]
+== Troubleshooting monitoring
+++++
+<titleabbrev>Troubleshooting</titleabbrev>
+++++
+
+Use the information in this section to troubleshoot common problems and find
+answers for frequently asked questions. See also
+{logstash-ref}/monitoring-troubleshooting.html[Troubleshooting monitoring in {ls}].
+
+For issues that you cannot fix yourself … we’re here to help.
+If you are an existing Elastic customer with a support contract, please create
+a ticket in the
+https://support.elastic.co/customers/s/login/[Elastic Support portal].
+Or post in the https://discuss.elastic.co/[Elastic forum].
+
+*Symptoms*:
+There is no information about your cluster on the *Stack Monitoring* page in
+{kib}.
+
+*Resolution*:
+Check whether the appropriate indices exist on the monitoring cluster. For
+example, use the <<cat-indices,cat indices>> command to verify that
+there is a `.monitoring-kibana*` index for your {kib} monitoring data and a
+`.monitoring-es*` index for your {es} monitoring data. If you are collecting
+monitoring data by using {metricbeat} the indices have `-mb` in their names. If
+the indices do not exist, review your configuration. For example, see
+<<monitoring-production>>.
+

docs/reference/settings/monitoring-settings.asciidoc

@@ -117,9 +117,9 @@ the `http` exporter will not be deleted automatically.
 Configures where the agent stores monitoring data. By default, the agent uses a
 local exporter that indexes monitoring data on the cluster where it is installed.
 Use an HTTP exporter to send data to a separate monitoring cluster. For more
-information, see <<local-exporter-settings,Local Exporter Settings>>,
-<<http-exporter-settings,HTTP Exporter Settings>>, and
-{xpack-ref}/how-monitoring-works.html[How Monitoring Works].
+information, see <<local-exporter-settings,Local exporter settings>>,
+<<http-exporter-settings,HTTP exporter settings>>, and
+<<how-monitoring-works>>.
 
 [float]
 [[local-exporter-settings]]