387 lines
12 KiB
Plaintext
387 lines
12 KiB
Plaintext
[role="xpack"]
|
|
[testenv="platinum"]
|
|
[[ccr-getting-started]]
|
|
=== Set up {ccr}
|
|
You can manually create follower indices to replicate specific indices on a
|
|
remote cluster, or configure auto-follow patterns to automatically create
|
|
follower indices for new time series.
|
|
|
|
After the follower index is created, the
|
|
<<ccr-remote-recovery, remote recovery>> process copies all of the Lucene
|
|
segment files from the remote cluster to the local cluster.
|
|
|
|
To set up {ccr}:
|
|
|
|
. <<ccr-getting-started-remote-cluster,Connect a local cluster to a remote cluster>>
|
|
. <<ccr-getting-started-leader-index,Identify the index (or time series indices) you want to replicate on the remote cluster>>
|
|
. <<ccr-enable-soft-deletes,Enable soft deletes on the leader index>>
|
|
. Manually create a follower index or create an auto-follow pattern:
|
|
* To replicate the leader index, <<ccr-getting-started-follower-index,manually create a follower index>>
|
|
* To automatically follow time series indices, <<ccr-getting-started-auto-follow,create an auto-follow pattern>>
|
|
|
|
[[ccr-getting-started-prerequisites]]
|
|
==== Prerequisites
|
|
If the Elastic {security-features} are enabled in your local and remote
|
|
clusters, you need a user with appropriate authority to complete the steps
|
|
in this tutorial.
|
|
|
|
By default, you can complete the following steps as the built-in
|
|
`elastic` user. However, you must <<get-started-built-in-users,set a password>>
|
|
for this user before proceeding.
|
|
|
|
WARNING: If you are performing these steps in a production environment, do
|
|
not use the `elastic` user.
|
|
|
|
Alternatively, you can assign the appropriate privileges to a user ID of your
|
|
choice. On the remote cluster that contains the leader index, a user must have
|
|
the `read_ccr` cluster privilege and `monitor` and `read` privileges on the
|
|
leader index.
|
|
|
|
[source,yml]
|
|
--------------------------------------------------
|
|
ccr_user:
|
|
cluster:
|
|
- read_ccr
|
|
indices:
|
|
- names: [ 'leader-index' ]
|
|
privileges:
|
|
- monitor
|
|
- read
|
|
--------------------------------------------------
|
|
|
|
On the local cluster that contains the follower index, the same user will need
|
|
the `manage_ccr` cluster privilege and `monitor`, `read`, `write` and
|
|
`manage_follow_index` privileges on the follower index.
|
|
|
|
[source,yml]
|
|
--------------------------------------------------
|
|
ccr_user:
|
|
cluster:
|
|
- manage_ccr
|
|
indices:
|
|
- names: [ 'follower-index' ]
|
|
privileges:
|
|
- monitor
|
|
- read
|
|
- write
|
|
- manage_follow_index
|
|
--------------------------------------------------
|
|
|
|
If you are managing
|
|
<<ccr-getting-started-remote-cluster,connecting to the remote cluster>> using
|
|
the cluster update settings API, you will also need a user with the `all`
|
|
cluster privilege.
|
|
|
|
[[ccr-getting-started-remote-cluster]]
|
|
==== Connect to a remote cluster
|
|
Connect your local cluster to a
|
|
<<modules-remote-clusters,remote cluster>> to begin using cross-cluster
|
|
replication.
|
|
|
|
To configure a {kibana-ref}/working-remote-clusters.html[remote cluster],
|
|
access {kib} and go to
|
|
*Management > Stack Management*. In the side navigation, select
|
|
*Remote Clusters*.
|
|
|
|
Add a remote cluster by specifying the IP address or host name, followed by the
|
|
transport port of the remote cluster.
|
|
|
|
[role="screenshot"]
|
|
image::images/ccr-add-remote-cluster.png["The Add remote clusters page in {kib}"]
|
|
|
|
[%collapsible]
|
|
.API example
|
|
====
|
|
Use the <<cluster-update-settings,cluster update settings API>> to add a remote cluster:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /_cluster/settings
|
|
{
|
|
"persistent" : {
|
|
"cluster" : {
|
|
"remote" : {
|
|
"leader" : {
|
|
"seeds" : [
|
|
"127.0.0.1:9300" <1>
|
|
]
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[setup:host]
|
|
// TEST[s/127.0.0.1:9300/\${transport_host}/]
|
|
<1> Specifies the hostname and transport port of a seed node in the remote
|
|
cluster.
|
|
|
|
You can verify that the local cluster is successfully connected to the remote
|
|
cluster.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET /_remote/info
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
The API will respond by showing that the local cluster is connected to the
|
|
remote cluster.
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"leader" : {
|
|
"seeds" : [
|
|
"127.0.0.1:9300"
|
|
],
|
|
"connected" : true, <1>
|
|
"num_nodes_connected" : 1, <2>
|
|
"max_connections_per_cluster" : 3,
|
|
"initial_connect_timeout" : "30s",
|
|
"skip_unavailable" : false,
|
|
"mode" : "sniff"
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[s/127.0.0.1:9300/$body.leader.seeds.0/]
|
|
// TEST[s/"connected" : true/"connected" : $body.leader.connected/]
|
|
// TEST[s/"num_nodes_connected" : 1/"num_nodes_connected" : $body.leader.num_nodes_connected/]
|
|
<1> This shows the local cluster is connected to the remote cluster with cluster
|
|
alias `leader`
|
|
<2> This shows the number of nodes in the remote cluster the local cluster is
|
|
connected to.
|
|
====
|
|
|
|
[[ccr-getting-started-leader-index]]
|
|
==== Create a leader index
|
|
To create a leader index, access {kib} on your _remote_ cluster and go to
|
|
*Management > Dev Tools*.
|
|
|
|
Copy the following example into the Console to create a leader index named
|
|
`server-metrics` in your remote cluster:
|
|
|
|
[%collapsible]
|
|
.Leader index example
|
|
====
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /server-metrics
|
|
{
|
|
"settings" : {
|
|
"index" : {
|
|
"number_of_shards" : 1,
|
|
"number_of_replicas" : 0
|
|
}
|
|
},
|
|
"mappings" : {
|
|
"properties" : {
|
|
"@timestamp" : {
|
|
"type" : "date"
|
|
},
|
|
"accept" : {
|
|
"type" : "long"
|
|
},
|
|
"deny" : {
|
|
"type" : "long"
|
|
},
|
|
"host" : {
|
|
"type" : "keyword"
|
|
},
|
|
"response" : {
|
|
"type" : "float"
|
|
},
|
|
"service" : {
|
|
"type" : "keyword"
|
|
},
|
|
"total" : {
|
|
"type" : "long"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
====
|
|
|
|
[[ccr-enable-soft-deletes]]
|
|
==== Enable soft deletes on leader indices
|
|
<<ccr-leader-requirements,Soft deletes>> must be enabled for indices that you want to
|
|
use as leader indices. Soft deletes are enabled by default on new indices
|
|
created on or after {es} 7.0.0, so
|
|
*no further action is required if your cluster is running {es} 7.0.0 or later*.
|
|
|
|
include::{es-ref-dir}/ccr/index.asciidoc[tag=ccr-existing-indices-tag]
|
|
|
|
To enable soft deletes on indices created on versions of
|
|
{es} between 6.5.0 and 7.0.0, set <<ccr-index-soft-deletes,`index.soft_deletes.enabled`>> to `true`.
|
|
|
|
[[ccr-getting-started-follower-index]]
|
|
==== Create a follower index
|
|
When you create a {kibana-ref}/managing-cross-cluster-replication.html#_create_specific_follower_indices[follower index], you
|
|
must reference the
|
|
<<ccr-getting-started-remote-cluster,remote cluster>> and the
|
|
<<ccr-getting-started-leader-index,leader index>> that you created in the remote
|
|
cluster.
|
|
|
|
To create a follower index, access {kib} and go to
|
|
*Management > Stack Management*. In the side navigation, select
|
|
*Cross-Cluster Replication* and choose the *Follower Indices* tab.
|
|
|
|
. Choose the remote cluster containing the index you want to replicate, which
|
|
is `leader` if you are following the tutorial.
|
|
. Enter the name of the leader index, which is `server-metrics` if you are
|
|
following the tutorial.
|
|
|
|
image::images/ccr-add-follower-index.png["Adding a follower index named server-metrics in {kib}"]
|
|
|
|
The follower index is initialized using the
|
|
<<ccr-remote-recovery, remote recovery>>
|
|
process, which transfers the existing Lucene segment files from the leader
|
|
index to the follower index. The index status changes to *Paused*. When the
|
|
remote recovery process is complete, the index following begins and the status
|
|
changes to *Active*.
|
|
|
|
When you index documents into your leader index, the documents are replicated
|
|
in the follower index.
|
|
|
|
[role="screenshot"]
|
|
image::images/ccr-follower-index.png["The Cross-Cluster Replication page in {kib}"]
|
|
|
|
[%collapsible]
|
|
.API example
|
|
====
|
|
Use the <<ccr-put-follow,create follower API>> to create follower indices.
|
|
When you create a follower index, you must reference the
|
|
<<ccr-getting-started-remote-cluster,remote cluster>> and the
|
|
<<ccr-getting-started-leader-index,leader index>> that you created in the
|
|
remote cluster.
|
|
|
|
When initiating the follower request, the response returns before the
|
|
<<ccr-remote-recovery, remote recovery>> process completes. To wait for the process
|
|
to complete, add the `wait_for_active_shards` parameter to your request.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /server-metrics-follower/_ccr/follow?wait_for_active_shards=1
|
|
{
|
|
"remote_cluster" : "leader",
|
|
"leader_index" : "server-metrics"
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
//////////////////////////
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"follow_index_created" : true,
|
|
"follow_index_shards_acked" : true,
|
|
"index_following_started" : true
|
|
}
|
|
--------------------------------------------------
|
|
|
|
//////////////////////////
|
|
|
|
Use the
|
|
<<ccr-get-follow-stats,get follower stats API>> to inspect the status of
|
|
replication
|
|
|
|
//////////////////////////
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST /server-metrics-follower/_ccr/pause_follow
|
|
|
|
POST /server-metrics-follower/_close
|
|
|
|
POST /server-metrics-follower/_ccr/unfollow
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
//////////////////////////
|
|
====
|
|
|
|
[[ccr-getting-started-auto-follow]]
|
|
==== Automatically create follower indices
|
|
Create <<ccr-auto-follow,auto-follow patterns>> to automatically follow time
|
|
series indices that are periodically created in a remote cluster (such as daily
|
|
{beats} indices).
|
|
|
|
With an auto-follow pattern, you reference the
|
|
<<ccr-getting-started-remote-cluster,remote cluster>> connected to your
|
|
local cluster. You must also specify a collection of patterns that match the
|
|
indices you want to automatically follow.
|
|
|
|
// tag::ccr-create-auto-follow-pattern-tag[]
|
|
To create follower indices from an {kibana-ref}/managing-cross-cluster-replication.html#_create_follower_indices_from_an_auto_follow_pattern[auto-follow pattern],
|
|
access {kib} on your remote cluster and go to
|
|
*Management > Stack Management*. In the side navigation, select
|
|
*Cross Cluster Replication* and choose the *Auto-follow patterns* tab.
|
|
|
|
[role="screenshot"]
|
|
image::images/auto-follow-patterns.png["The Auto-follow patterns page in {kib}"]
|
|
|
|
* Enter a name for the auto-follow pattern. For this tutorial, enter `beats`
|
|
as the name.
|
|
* Choose the remote cluster containing the index you want to replicate, which
|
|
is `leader` if you are following the tutorial.
|
|
* Enter one or more index patterns that identify the indices you want to
|
|
replicate from the remote cluster. For this tutorial, enter
|
|
`metricbeat-*,packetbeat-*` as the index pattern.
|
|
* Enter *copy-* as the prefix to apply to the names of the follower indices so
|
|
you can more easily identify replicated indices.
|
|
|
|
As new indices matching these patterns are
|
|
created, they are replicated to the follower indices.
|
|
|
|
// end::ccr-create-auto-follow-pattern-tag[]
|
|
|
|
[%collapsible]
|
|
.API example
|
|
====
|
|
Use the <<ccr-put-auto-follow-pattern,create auto-follow pattern API>> to
|
|
configure auto-follow patterns.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /_ccr/auto_follow/beats
|
|
{
|
|
"remote_cluster" : "leader",
|
|
"leader_index_patterns" :
|
|
[
|
|
"metricbeat-*", <1>
|
|
"packetbeat-*" <2>
|
|
],
|
|
"follow_index_pattern" : "{{leader_index}}-copy" <3>
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
<1> Automatically follow new {metricbeat} indices.
|
|
<2> Automatically follow new {packetbeat} indices.
|
|
<3> The name of the follower index is derived from the name of the leader index
|
|
by adding the suffix `-copy` to the name of the leader index.
|
|
|
|
//////////////////////////
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"acknowledged" : true
|
|
}
|
|
--------------------------------------------------
|
|
|
|
//////////////////////////
|
|
|
|
//////////////////////////
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
DELETE /_ccr/auto_follow/beats
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
//////////////////////////
|
|
====
|