OpenSearch/docs/en/monitoring/exporters.asciidoc

[role="xpack"]
[[es-monitoring-exporters]]
=== Exporters

The purpose of exporters is to take data collected from any Elastic Stack
source and route it to the monitoring cluster. It is possible to configure
more than one exporter, but the general and default setup is to use a single
exporter.

There are two types of exporters in {es}:

`local`::
The default exporter used by {monitoring} for {es}. This exporter routes data
back into the _same_ cluster.

`http`::
The preferred exporter, which you can use to route data into any supported
{es} cluster accessible via HTTP. Production environments should always use a
separate monitoring cluster. For more information, see
{xpack-ref}/monitoring-production.html[Monitoring in a production environment].

Both exporters serve the same purpose: to set up the monitoring cluster and route
monitoring data. However, they perform these tasks in very different ways. Even
though things happen differently, both exporters are capable of sending all of
the same data.

Exporters are configurable at both the node and cluster level. Cluster-wide
settings, which are updated with the
<<cluster-update-settings,`_cluster/settings` API>>, take precedence over
settings in the `elasticsearch.yml` file on each node. When you update an
exporter, it is completely replaced by the updated version of the exporter.

IMPORTANT: It is critical that all nodes share the same setup. Otherwise,
monitoring data might be routed in different ways or to different places.

When the exporters route monitoring data into the monitoring cluster, they use
`_bulk` indexing for optimal performance. There is no queuing--in memory or
persisted to disk--so any failure during the export results in the loss of
that batch of monitoring data. This design limits the impact on {es} and the
assumption is that the next pass will succeed.

[[es-monitoring-default-exporter]]
==== Default exporters

If a node or cluster does not explicitly define an {monitoring} exporter, the
following default exporter is used:

[source,yaml]
---------------------------------------------------
xpack.monitoring.exporters.default_local: <1>
  type: local
---------------------------------------------------
<1> The exporter name uniquely defines the exporter, but it is otherwise unused.
    When you specify your own exporters, you do not need to explicitly overwrite
    or reference `default_local`.

If another exporter is already defined, the default exporter is _not_ created.
When you define a new exporter, if the default exporter exists, it is
automatically removed.

[[es-monitoring-templates]]
==== Exporter templates and ingest pipelines

Before exporters can route monitoring data, they must set up certain {es}
resources. These resources include templates and ingest pipelines. The
following table lists the templates that are required before an exporter can
route monitoring data:

[options="header"]
|=======================
| Template               | Purpose
| `.monitoring-alerts`   | All cluster alerts for monitoring data.
| `.monitoring-beats`    | All Beats monitoring data.
| `.monitoring-es`       | All {es} monitoring data.
| `.monitoring-kibana`   | All {kib} monitoring data.
| `.monitoring-logstash` | All Logstash monitoring data.
|=======================

The templates are ordinary {es} templates that control the default settings and
mappings for the monitoring indices.

By default, monitoring indices are created daily (for example,
`.monitoring-es-6-2017.08.26`). You can change the default date suffix for
monitoring indices with the `index.name.time_format` setting. You can use this
setting to control how frequently monitoring indices are created by a specific
`http` exporter. You cannot use this setting with `local` exporters. For more
information, see <<http-exporter-settings>>.

WARNING: Some users create their own templates that match _all_ index patterns,
which therefore impact the monitoring indices that get created. It is critical
that you do not disable `_source` storage for the monitoring indices. If you do,
{monitoring} for {kib} does not work and you cannot visualize monitoring data
for your cluster.

The following table lists the ingest pipelines that are required before an
exporter can route monitoring data:

[options="header"]
|=======================
| Pipeline               | Purpose
| `xpack_monitoring_2`   | Upgrades X-Pack monitoring data coming from X-Pack
5.0 - 5.4 to be compatible with the format used in {monitoring} 5.5.
| `xpack_monitoring_6`   | A placeholder pipeline that is empty.
|=======================

Exporters handle the setup of these resources before ever sending data. If
resource setup fails (for example, due to security permissions), no data is sent
and warnings are logged.

NOTE: Empty pipelines are evaluated on the coordinating node during indexing and
they are ignored without any extra effort. This inherently makes them a safe,
no-op operation.

For monitoring clusters that have disabled `node.ingest` on all nodes, it is
possible to disable the use of the ingest pipeline feature. However, doing so
blocks its purpose, which is to upgrade older monitoring data as our mappings
improve over time. Beginning in 6.0, the ingest pipeline feature is a
requirement on the monitoring cluster; you must have `node.ingest` enabled on at
least one node.

WARNING: Once any node running 5.5 or later has set up the templates and ingest
pipeline on a monitoring cluster, you must use {kib} 5.5 or later to view all
subsequent data on the monitoring cluster. The easiest way to determine
whether this update has occurred is by checking for the presence of indices
matching `.monitoring-es-6-*` (or more concretely the existence of the
new pipeline). Versions prior to 5.5 used `.monitoring-es-2-*`.