[role="xpack"] [testenv="platinum"] [[ccr-getting-started]] === Set up {ccr} You can manually create follower indices to replicate specific indices on a remote cluster, or configure auto-follow patterns to automatically create follower indices for new time series. After the follower index is created, the <> process copies all of the Lucene segment files from the remote cluster to the local cluster. To set up {ccr}: . <> . <> . <> . Manually create a follower index or create an auto-follow pattern: * To replicate the leader index, <> * To automatically follow time series indices, <> [[ccr-getting-started-prerequisites]] ==== Prerequisites If the Elastic {security-features} are enabled in your local and remote clusters, you need a user with appropriate authority to complete the steps in this tutorial. By default, you can complete the following steps as the built-in `elastic` user. However, you must <> for this user before proceeding. WARNING: If you are performing these steps in a production environment, do not use the `elastic` user. Alternatively, you can assign the appropriate privileges to a user ID of your choice. On the remote cluster that contains the leader index, a user must have the `read_ccr` cluster privilege and `monitor` and `read` privileges on the leader index. [source,yml] -------------------------------------------------- ccr_user: cluster: - read_ccr indices: - names: [ 'leader-index' ] privileges: - monitor - read -------------------------------------------------- On the local cluster that contains the follower index, the same user will need the `manage_ccr` cluster privilege and `monitor`, `read`, `write` and `manage_follow_index` privileges on the follower index. [source,yml] -------------------------------------------------- ccr_user: cluster: - manage_ccr indices: - names: [ 'follower-index' ] privileges: - monitor - read - write - manage_follow_index -------------------------------------------------- If you are managing <> using the cluster update settings API, you will also need a user with the `all` cluster privilege. [[ccr-getting-started-remote-cluster]] ==== Connect to a remote cluster Connect your local cluster to a <> to begin using cross-cluster replication. To configure a {kibana-ref}/working-remote-clusters.html[remote cluster], access {kib} and go to *Management > Stack Management*. In the side navigation, select *Remote Clusters*. Add a remote cluster by specifying the IP address or host name, followed by the transport port of the remote cluster. [role="screenshot"] image::images/ccr-add-remote-cluster.png["The Add remote clusters page in {kib}"] [%collapsible] .API example ==== Use the <> to add a remote cluster: [source,console] -------------------------------------------------- PUT /_cluster/settings { "persistent" : { "cluster" : { "remote" : { "leader" : { "seeds" : [ "127.0.0.1:9300" <1> ] } } } } } -------------------------------------------------- // TEST[setup:host] // TEST[s/127.0.0.1:9300/\${transport_host}/] <1> Specifies the hostname and transport port of a seed node in the remote cluster. You can verify that the local cluster is successfully connected to the remote cluster. [source,console] -------------------------------------------------- GET /_remote/info -------------------------------------------------- // TEST[continued] The API will respond by showing that the local cluster is connected to the remote cluster. [source,console-result] -------------------------------------------------- { "leader" : { "seeds" : [ "127.0.0.1:9300" ], "connected" : true, <1> "num_nodes_connected" : 1, <2> "max_connections_per_cluster" : 3, "initial_connect_timeout" : "30s", "skip_unavailable" : false, "mode" : "sniff" } } -------------------------------------------------- // TESTRESPONSE[s/127.0.0.1:9300/$body.leader.seeds.0/] // TEST[s/"connected" : true/"connected" : $body.leader.connected/] // TEST[s/"num_nodes_connected" : 1/"num_nodes_connected" : $body.leader.num_nodes_connected/] <1> This shows the local cluster is connected to the remote cluster with cluster alias `leader` <2> This shows the number of nodes in the remote cluster the local cluster is connected to. ==== [[ccr-getting-started-leader-index]] ==== Create a leader index To create a leader index, access {kib} on your _remote_ cluster and go to *Management > Dev Tools*. Copy the following example into the Console to create a leader index named `server-metrics` in your remote cluster: [%collapsible] .Leader index example ==== [source,console] -------------------------------------------------- PUT /server-metrics { "settings" : { "index" : { "number_of_shards" : 1, "number_of_replicas" : 0 } }, "mappings" : { "properties" : { "@timestamp" : { "type" : "date" }, "accept" : { "type" : "long" }, "deny" : { "type" : "long" }, "host" : { "type" : "keyword" }, "response" : { "type" : "float" }, "service" : { "type" : "keyword" }, "total" : { "type" : "long" } } } } -------------------------------------------------- // TEST[continued] ==== [[ccr-enable-soft-deletes]] ==== Enable soft deletes on leader indices <> must be enabled for indices that you want to use as leader indices. Soft deletes are enabled by default on new indices created on or after {es} 7.0.0, so *no further action is required if your cluster is running {es} 7.0.0 or later*. include::{es-ref-dir}/ccr/index.asciidoc[tag=ccr-existing-indices-tag] To enable soft deletes on indices created on versions of {es} between 6.5.0 and 7.0.0, set <> to `true`. [[ccr-getting-started-follower-index]] ==== Create a follower index When you create a {kibana-ref}/managing-cross-cluster-replication.html#_create_specific_follower_indices[follower index], you must reference the <> and the <> that you created in the remote cluster. To create a follower index, access {kib} and go to *Management > Stack Management*. In the side navigation, select *Cross-Cluster Replication* and choose the *Follower Indices* tab. . Choose the remote cluster containing the index you want to replicate, which is `leader` if you are following the tutorial. . Enter the name of the leader index, which is `server-metrics` if you are following the tutorial. image::images/ccr-add-follower-index.png["Adding a follower index named server-metrics in {kib}"] The follower index is initialized using the <> process, which transfers the existing Lucene segment files from the leader index to the follower index. The index status changes to *Paused*. When the remote recovery process is complete, the index following begins and the status changes to *Active*. When you index documents into your leader index, the documents are replicated in the follower index. [role="screenshot"] image::images/ccr-follower-index.png["The Cross-Cluster Replication page in {kib}"] [%collapsible] .API example ==== Use the <> to create follower indices. When you create a follower index, you must reference the <> and the <> that you created in the remote cluster. When initiating the follower request, the response returns before the <> process completes. To wait for the process to complete, add the `wait_for_active_shards` parameter to your request. [source,console] -------------------------------------------------- PUT /server-metrics-follower/_ccr/follow?wait_for_active_shards=1 { "remote_cluster" : "leader", "leader_index" : "server-metrics" } -------------------------------------------------- // TEST[continued] ////////////////////////// [source,console-result] -------------------------------------------------- { "follow_index_created" : true, "follow_index_shards_acked" : true, "index_following_started" : true } -------------------------------------------------- ////////////////////////// Use the <> to inspect the status of replication ////////////////////////// [source,console] -------------------------------------------------- POST /server-metrics-follower/_ccr/pause_follow POST /server-metrics-follower/_close POST /server-metrics-follower/_ccr/unfollow -------------------------------------------------- // TEST[continued] ////////////////////////// ==== [[ccr-getting-started-auto-follow]] ==== Automatically create follower indices Create <> to automatically follow time series indices that are periodically created in a remote cluster (such as daily {beats} indices). With an auto-follow pattern, you reference the <> connected to your local cluster. You must also specify a collection of patterns that match the indices you want to automatically follow. // tag::ccr-create-auto-follow-pattern-tag[] To create follower indices from an {kibana-ref}/managing-cross-cluster-replication.html#_create_follower_indices_from_an_auto_follow_pattern[auto-follow pattern], access {kib} on your remote cluster and go to *Management > Stack Management*. In the side navigation, select *Cross Cluster Replication* and choose the *Auto-follow patterns* tab. [role="screenshot"] image::images/auto-follow-patterns.png["The Auto-follow patterns page in {kib}"] * Enter a name for the auto-follow pattern. For this tutorial, enter `beats` as the name. * Choose the remote cluster containing the index you want to replicate, which is `leader` if you are following the tutorial. * Enter one or more index patterns that identify the indices you want to replicate from the remote cluster. For this tutorial, enter `metricbeat-*,packetbeat-*` as the index pattern. * Enter *copy-* as the prefix to apply to the names of the follower indices so you can more easily identify replicated indices. As new indices matching these patterns are created, they are replicated to the follower indices. // end::ccr-create-auto-follow-pattern-tag[] [%collapsible] .API example ==== Use the <> to configure auto-follow patterns. [source,console] -------------------------------------------------- PUT /_ccr/auto_follow/beats { "remote_cluster" : "leader", "leader_index_patterns" : [ "metricbeat-*", <1> "packetbeat-*" <2> ], "follow_index_pattern" : "{{leader_index}}-copy" <3> } -------------------------------------------------- // TEST[continued] <1> Automatically follow new {metricbeat} indices. <2> Automatically follow new {packetbeat} indices. <3> The name of the follower index is derived from the name of the leader index by adding the suffix `-copy` to the name of the leader index. ////////////////////////// [source,console-result] -------------------------------------------------- { "acknowledged" : true } -------------------------------------------------- ////////////////////////// ////////////////////////// [source,console] -------------------------------------------------- DELETE /_ccr/auto_follow/beats -------------------------------------------------- // TEST[continued] ////////////////////////// ====