334 lines
11 KiB
Plaintext
334 lines
11 KiB
Plaintext
[role="xpack"]
|
||
[testenv="platinum"]
|
||
[[ccr-getting-started]]
|
||
=== Tutorial: Set up {ccr}
|
||
++++
|
||
<titleabbrev>Set up {ccr}</titleabbrev>
|
||
++++
|
||
|
||
////
|
||
[source,console]
|
||
----
|
||
PUT /server-metrics
|
||
{
|
||
"settings" : {
|
||
"index" : {
|
||
"number_of_shards" : 1,
|
||
"number_of_replicas" : 0
|
||
}
|
||
},
|
||
"mappings" : {
|
||
"properties" : {
|
||
"@timestamp" : {
|
||
"type" : "date"
|
||
},
|
||
"accept" : {
|
||
"type" : "long"
|
||
},
|
||
"deny" : {
|
||
"type" : "long"
|
||
},
|
||
"host" : {
|
||
"type" : "keyword"
|
||
},
|
||
"response" : {
|
||
"type" : "float"
|
||
},
|
||
"service" : {
|
||
"type" : "keyword"
|
||
},
|
||
"total" : {
|
||
"type" : "long"
|
||
}
|
||
}
|
||
}
|
||
}
|
||
----
|
||
// TESTSETUP
|
||
////
|
||
|
||
Use this guide to set up {ccr} (CCR) between clusters in two
|
||
datacenters. Replicating your data across datacenters provides several benefits:
|
||
|
||
* Brings data closer to your users or application server to reduce latency and
|
||
response time
|
||
* Provides your mission-critical applications with the tolerance to withstand datacenter or region outages
|
||
|
||
In this guide, you'll learn how to:
|
||
|
||
* Configure a <<modules-remote-clusters,remote cluster>> with a leader index
|
||
* Create a follower index on a local cluster
|
||
* Create an auto-follow pattern to automatically follow time series indices
|
||
that are periodically created in a remote cluster
|
||
|
||
You can manually create follower indices to replicate specific indices on a
|
||
remote cluster, or configure auto-follow patterns to replicate rolling time series indices.
|
||
|
||
video::https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt994089f5e841ad69/5f6265de6f40ab4648b5cf9b/ccr-setup-video-edited.mp4[width=700, height=500, options="autoplay,loop"]
|
||
|
||
[[ccr-getting-started-prerequisites]]
|
||
==== Prerequisites
|
||
To complete this tutorial, you need:
|
||
|
||
* A license on both clusters that includes {ccr}. {kibana-ref}/managing-licenses.html[Activate a free 30-day trial].
|
||
* The `read_ccr` cluster privilege and `monitor` and `read` privileges
|
||
for the leader index on the remote cluster. <<stack-management-ccr-remote,Configure remote cluster privileges>>.
|
||
* The `manage_ccr` cluster privilege and `monitor`, `read`, `write` and
|
||
`manage_follow_index` privileges to configure remote clusters and follower
|
||
indices on the local cluster. <<stack-management-ccr-local,Configure local cluster privileges>>.
|
||
* An index on the remote cluster that contains the data you want to replicate.
|
||
This tutorial uses the sample eCommerce orders data set.
|
||
{kibana-ref}/get-started.html#gs-get-data-into-kibana[Load sample data].
|
||
|
||
[[ccr-getting-started-remote-cluster]]
|
||
==== Connect to a remote cluster
|
||
To replicate an index on a remote cluster (Cluster A) to a local cluster (Cluster B), you configure Cluster A as a remote on Cluster B.
|
||
|
||
image::images/ccr-tutorial-clusters.png[ClusterA contains the leader index and ClusterB contains the follower index]
|
||
|
||
To configure a remote cluster from Stack Management in {kib}:
|
||
|
||
. Select *Remote Clusters* from the side navigation.
|
||
. Specify the IP address or host name of the remote cluster (ClusterB),
|
||
followed by the transport port of the remote cluster (defaults to `9300`). For
|
||
example, `192.168.1.1:9300`.
|
||
|
||
[role="screenshot"]
|
||
image::images/ccr-add-remote-cluster.png["The Add remote clusters page in {kib}"]
|
||
|
||
[%collapsible]
|
||
.API example
|
||
====
|
||
Use the <<cluster-update-settings,cluster update settings API>> to add a remote cluster:
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
PUT /_cluster/settings
|
||
{
|
||
"persistent" : {
|
||
"cluster" : {
|
||
"remote" : {
|
||
"leader" : {
|
||
"seeds" : [
|
||
"127.0.0.1:9300" <1>
|
||
]
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
// TEST[setup:host]
|
||
// TEST[s/127.0.0.1:9300/\${transport_host}/]
|
||
<1> Specifies the hostname and transport port of a seed node in the remote
|
||
cluster.
|
||
|
||
You can verify that the local cluster is successfully connected to the remote
|
||
cluster.
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
GET /_remote/info
|
||
--------------------------------------------------
|
||
// TEST[continued]
|
||
|
||
The API will respond by showing that the local cluster is connected to the
|
||
remote cluster.
|
||
|
||
[source,console-result]
|
||
--------------------------------------------------
|
||
{
|
||
"leader" : {
|
||
"seeds" : [
|
||
"127.0.0.1:9300"
|
||
],
|
||
"connected" : true, <1>
|
||
"num_nodes_connected" : 1, <2>
|
||
"max_connections_per_cluster" : 3,
|
||
"initial_connect_timeout" : "30s",
|
||
"skip_unavailable" : false,
|
||
"mode" : "sniff"
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
// TESTRESPONSE[s/127.0.0.1:9300/$body.leader.seeds.0/]
|
||
// TEST[s/"connected" : true/"connected" : $body.leader.connected/]
|
||
// TEST[s/"num_nodes_connected" : 1/"num_nodes_connected" : $body.leader.num_nodes_connected/]
|
||
<1> This shows the local cluster is connected to the remote cluster with cluster
|
||
alias `leader`
|
||
<2> This shows the number of nodes in the remote cluster the local cluster is
|
||
connected to.
|
||
====
|
||
|
||
[[ccr-enable-soft-deletes]]
|
||
==== Enable soft deletes on leader indices
|
||
To follow an index, it must have been created with
|
||
<<ccr-leader-requirements,soft deletes>> enabled. If the index doesn’t have
|
||
soft deletes enabled, you must reindex it and use the new index as the leader
|
||
index. Soft deletes are enabled by default on new indices
|
||
created with {es} 7.0.0 and later.
|
||
|
||
[[ccr-getting-started-follower-index]]
|
||
==== Create a follower index to replicate a specific index
|
||
When you create a follower index, you reference the remote cluster and the
|
||
leader index in your remote cluster.
|
||
|
||
To create a follower index from Stack Management in {kib}:
|
||
|
||
. Select *Cross-Cluster Replication* in the side navigation and choose the
|
||
*Follower Indices* tab.
|
||
. Choose the cluster (ClusterA) containing the leader index you want to
|
||
replicate.
|
||
. Enter the name of the leader index, which is
|
||
`kibana_sample_data_ecommerce` if you are following the tutorial.
|
||
. Enter a name for your follower index, such as `follower-kibana-sample-data`.
|
||
|
||
image::images/ccr-add-follower-index.png["Adding a follower index named server-metrics in {kib}"]
|
||
|
||
{es} initializes the follower using the
|
||
<<ccr-remote-recovery, remote recovery>>
|
||
process, which transfers the existing Lucene segment files from the leader
|
||
index to the follower index. The index status changes to *Paused*. When the
|
||
remote recovery process is complete, the index following begins and the status
|
||
changes to *Active*.
|
||
|
||
When you index documents into your leader index, {es} replicates the documents
|
||
in the follower index.
|
||
|
||
[role="screenshot"]
|
||
image::images/ccr-follower-index.png["The Cross-Cluster Replication page in {kib}"]
|
||
|
||
[%collapsible]
|
||
.API example
|
||
====
|
||
Use the <<ccr-put-follow,create follower API>> to create follower indices.
|
||
When you create a follower index, you must reference the remote cluster and the
|
||
leader index that you created in the
|
||
remote cluster.
|
||
|
||
When initiating the follower request, the response returns before the
|
||
<<ccr-remote-recovery, remote recovery>> process completes. To wait for the process
|
||
to complete, add the `wait_for_active_shards` parameter to your request.
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
PUT /server-metrics-follower/_ccr/follow?wait_for_active_shards=1
|
||
{
|
||
"remote_cluster" : "leader",
|
||
"leader_index" : "server-metrics"
|
||
}
|
||
--------------------------------------------------
|
||
// TEST[continued]
|
||
|
||
//////////////////////////
|
||
|
||
[source,console-result]
|
||
--------------------------------------------------
|
||
{
|
||
"follow_index_created" : true,
|
||
"follow_index_shards_acked" : true,
|
||
"index_following_started" : true
|
||
}
|
||
--------------------------------------------------
|
||
|
||
//////////////////////////
|
||
|
||
Use the
|
||
<<ccr-get-follow-stats,get follower stats API>> to inspect the status of
|
||
replication
|
||
|
||
//////////////////////////
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
POST /server-metrics-follower/_ccr/pause_follow
|
||
|
||
POST /server-metrics-follower/_close
|
||
|
||
POST /server-metrics-follower/_ccr/unfollow
|
||
--------------------------------------------------
|
||
// TEST[continued]
|
||
|
||
//////////////////////////
|
||
====
|
||
|
||
[[ccr-getting-started-auto-follow]]
|
||
==== Create an auto-follow pattern to replicate time series indices
|
||
You use <<ccr-auto-follow,auto-follow patterns>> to automatically create new
|
||
followers for rolling time series indices. Whenever the name of a new index on
|
||
the remote cluster matches the auto-follow pattern, a corresponding follower
|
||
index is added to the local cluster.
|
||
|
||
An auto-follow pattern specifies the remote cluster you want to replicate from,
|
||
and one or more index patterns that specify the rolling time series indices you
|
||
want to replicate.
|
||
|
||
// tag::ccr-create-auto-follow-pattern-tag[]
|
||
To create an auto-follow pattern from Stack Management in {kib}:
|
||
|
||
. Select *Cross Cluster Replication* in the side navigation and choose the
|
||
*Auto-follow patterns* tab.
|
||
. Enter a name for the auto-follow pattern, such as `beats`.
|
||
. Choose the remote cluster that contains the index you want to replicate,
|
||
which in the example scenario is Cluster A.
|
||
. Enter one or more index patterns that identify the indices you want to
|
||
replicate from the remote cluster. For example, enter
|
||
`metricbeat-* packetbeat-*` to automatically create followers for {metricbeat} and {packetbeat} indices.
|
||
. Enter *follower-* as the prefix to apply to the names of the follower indices so
|
||
you can more easily identify replicated indices.
|
||
|
||
As new indices matching these patterns are
|
||
created on the remote, {es} automatically replicates them to local follower indices.
|
||
|
||
[role="screenshot"]
|
||
image::images/auto-follow-patterns.png["The Auto-follow patterns page in {kib}"]
|
||
|
||
// end::ccr-create-auto-follow-pattern-tag[]
|
||
|
||
[%collapsible]
|
||
.API example
|
||
====
|
||
Use the <<ccr-put-auto-follow-pattern,create auto-follow pattern API>> to
|
||
configure auto-follow patterns.
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
PUT /_ccr/auto_follow/beats
|
||
{
|
||
"remote_cluster" : "leader",
|
||
"leader_index_patterns" :
|
||
[
|
||
"metricbeat-*", <1>
|
||
"packetbeat-*" <2>
|
||
],
|
||
"follow_index_pattern" : "{{leader_index}}-copy" <3>
|
||
}
|
||
--------------------------------------------------
|
||
// TEST[continued]
|
||
<1> Automatically follow new {metricbeat} indices.
|
||
<2> Automatically follow new {packetbeat} indices.
|
||
<3> The name of the follower index is derived from the name of the leader index
|
||
by adding the suffix `-copy` to the name of the leader index.
|
||
|
||
//////////////////////////
|
||
|
||
[source,console-result]
|
||
--------------------------------------------------
|
||
{
|
||
"acknowledged" : true
|
||
}
|
||
--------------------------------------------------
|
||
|
||
//////////////////////////
|
||
|
||
//////////////////////////
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
DELETE /_ccr/auto_follow/beats
|
||
--------------------------------------------------
|
||
// TEST[continued]
|
||
|
||
//////////////////////////
|
||
====
|