Introduce CCR getting started guide (#35434)
This commit introduces a basic getting started guide for cross-cluster replication to the docs. Co-authored-by: "lcawl <lcawley@elastic.co>"
This commit is contained in:
Jason Tedor
GitHub
parent
76cefb06e3
commit
92fef40dbf
|
|
@ -1,7 +1,329 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="platinum"]
|
[testenv="platinum"]
|
||||||
[[ccr-getting-started]]
|
[[ccr-getting-started]]
|
||||||
== Getting Started
|
== Getting Started with {ccr}
|
||||||
|
|
||||||
beta[]
|
beta[]
|
||||||
This is the getting started section of the {ccr} docs.
|
|
||||||
|
This getting-started guide for {ccr} shows you how to:
|
||||||
|
|
||||||
|
* <<ccr-getting-started-remote-cluster,Connect a local cluster to a remote
|
||||||
|
cluster>>
|
||||||
|
* <<ccr-getting-started-leader-index,Create a leader index>> in a remote cluster
|
||||||
|
* <<ccr-getting-started-follower-index,Create a follower index>> that replicates
|
||||||
|
a leader index
|
||||||
|
* <<ccr-getting-started-auto-follow,Automatically create follower indices>>
|
||||||
|
|
||||||
|
[float]
|
||||||
|
[[ccr-getting-started-before-you-begin]]
|
||||||
|
=== Before you begin
|
||||||
|
. {stack-gs}/get-started-elastic-stack.html#install-elasticsearch[Install {es}]
|
||||||
|
on your local and remote clusters.
|
||||||
|
|
||||||
|
. Obtain a license that includes the {ccr} features. See
|
||||||
|
https://www.elastic.co/subscriptions[subscriptions] and
|
||||||
|
<<license-management>>.
|
||||||
|
|
||||||
|
. If the Elastic {security-features} are enabled in your local and remote
|
||||||
|
clusters, you need a user that has appropriate authority to perform the steps
|
||||||
|
in this tutorial.
|
||||||
|
+
|
||||||
|
--
|
||||||
|
[[ccr-getting-started-security]]
|
||||||
|
The {ccr} features use cluster privileges and built-in roles to make it easier
|
||||||
|
to control which users have authority to manage {ccr}.
|
||||||
|
|
||||||
|
By default, you can perform all of the steps in this tutorial by
|
||||||
|
using the built-in `elastic` user. However, a password must be set for this user
|
||||||
|
before the user can do anything. For information about how to set that password,
|
||||||
|
see <<security-getting-started>>.
|
||||||
|
|
||||||
|
If you are performing these steps in a production environment, take extra care
|
||||||
|
because the `elastic` user has the `superuser` role and you could inadvertently
|
||||||
|
make significant changes.
|
||||||
|
|
||||||
|
Alternatively, you can assign the appropriate privileges to a user ID of your
|
||||||
|
choice. On the remote cluster that contains the leader index, a user will need
|
||||||
|
the `read_ccr` cluster privilege and `monitor` and `read` privileges on the
|
||||||
|
leader index.
|
||||||
|
|
||||||
|
[source,yml]
|
||||||
|
--------------------------------------------------
|
||||||
|
ccr_user:
|
||||||
|
cluster:
|
||||||
|
- read_ccr
|
||||||
|
indices:
|
||||||
|
- names: [ 'leader-index' ]
|
||||||
|
privileges:
|
||||||
|
- monitor
|
||||||
|
- read
|
||||||
|
--------------------------------------------------
|
||||||
|
|
||||||
|
On the local cluster that contains the follower index, the same user will need
|
||||||
|
the `manage_ccr` cluster privilege and `monitor`, `read`, `write` and
|
||||||
|
`manage_follow_index` privileges on the follower index.
|
||||||
|
|
||||||
|
[source,yml]
|
||||||
|
--------------------------------------------------
|
||||||
|
ccr_user:
|
||||||
|
cluster:
|
||||||
|
- manage_ccr
|
||||||
|
indices:
|
||||||
|
- names: [ 'follower-index' ]
|
||||||
|
privileges:
|
||||||
|
- monitor
|
||||||
|
- read
|
||||||
|
- write
|
||||||
|
- manage_follow_index
|
||||||
|
--------------------------------------------------
|
||||||
|
|
||||||
|
If you are managing
|
||||||
|
<<ccr-getting-started-remote-cluster,connecting to the remote cluster>> via the
|
||||||
|
cluster update settings API, you will also need a user with the `all` cluster
|
||||||
|
privilege.
|
||||||
|
--
|
||||||
|
|
||||||
|
[float]
|
||||||
|
[[ccr-getting-started-remote-cluster]]
|
||||||
|
=== Connecting to a remote cluster
|
||||||
|
|
||||||
|
The {ccr} features require that you
|
||||||
|
{ref}/modules-remote-clusters.html[connect your local cluster to a remote
|
||||||
|
cluster]. In this tutorial, we will connect our local cluster to a remote
|
||||||
|
cluster with the cluster alias `leader`.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT /_cluster/settings
|
||||||
|
{
|
||||||
|
"persistent" : {
|
||||||
|
"cluster" : {
|
||||||
|
"remote" : {
|
||||||
|
"leader" : {
|
||||||
|
"seeds" : [
|
||||||
|
"127.0.0.1:9300" <1>
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[setup:host]
|
||||||
|
// TEST[s/127.0.0.1:9300/\${transport_host}/]
|
||||||
|
<1> Specifies the hostname and transport port of a seed node in the remote
|
||||||
|
cluster.
|
||||||
|
|
||||||
|
You can verify that the local cluster is successfully connected to the remote
|
||||||
|
cluster.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
GET /_remote/info
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[continued]
|
||||||
|
|
||||||
|
The API will respond by showing that the local cluster is connected to the
|
||||||
|
remote cluster.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
{
|
||||||
|
"leader" : {
|
||||||
|
"seeds" : [
|
||||||
|
"127.0.0.1:9300"
|
||||||
|
],
|
||||||
|
"connected" : true, <1>
|
||||||
|
"num_nodes_connected" : 1, <2>
|
||||||
|
"max_connections_per_cluster" : 3,
|
||||||
|
"initial_connect_timeout" : "30s",
|
||||||
|
"skip_unavailable" : false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// TESTRESPONSE
|
||||||
|
// TEST[s/127.0.0.1:9300/$body.leader.seeds.0/]
|
||||||
|
// TEST[s/"connected" : true/"connected" : $body.leader.connected/]
|
||||||
|
// TEST[s/"num_nodes_connected" : 1/"num_nodes_connected" : $body.leader.num_nodes_connected/]
|
||||||
|
<1> This shows the local cluster is connected to the remote cluster with cluster
|
||||||
|
alias `leader`
|
||||||
|
<2> This shows the number of nodes in the remote cluster the local cluster is
|
||||||
|
connected to.
|
||||||
|
|
||||||
|
[float]
|
||||||
|
[[ccr-getting-started-leader-index]]
|
||||||
|
=== Creating a leader index
|
||||||
|
|
||||||
|
Leader indices require special index settings to ensure that the operations that
|
||||||
|
need to be replicated are available when the
|
||||||
|
follower requests them from the leader. These settings are used to enable soft
|
||||||
|
deletes on the leader index and to control how many soft deletes are retained. A
|
||||||
|
_soft delete_ occurs whenever a document is deleted or updated. Soft deletes can
|
||||||
|
be enabled only on new indices created on or after {es} 6.5.0.
|
||||||
|
|
||||||
|
In the following example, we will create a leader index in the remote cluster:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT /server-metrics
|
||||||
|
{
|
||||||
|
"settings" : {
|
||||||
|
"index" : {
|
||||||
|
"number_of_shards" : 1,
|
||||||
|
"number_of_replicas" : 0,
|
||||||
|
"soft_deletes" : {
|
||||||
|
"enabled" : true, <1>
|
||||||
|
"retention" : {
|
||||||
|
"operations" : 1024 <2>
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"mappings" : {
|
||||||
|
"metric" : {
|
||||||
|
"properties" : {
|
||||||
|
"@timestamp" : {
|
||||||
|
"type" : "date"
|
||||||
|
},
|
||||||
|
"accept" : {
|
||||||
|
"type" : "long"
|
||||||
|
},
|
||||||
|
"deny" : {
|
||||||
|
"type" : "long"
|
||||||
|
},
|
||||||
|
"host" : {
|
||||||
|
"type" : "keyword"
|
||||||
|
},
|
||||||
|
"response" : {
|
||||||
|
"type" : "float"
|
||||||
|
},
|
||||||
|
"service" : {
|
||||||
|
"type" : "keyword"
|
||||||
|
},
|
||||||
|
"total" : {
|
||||||
|
"type" : "long"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[continued]
|
||||||
|
<1> Enables soft deletes on the leader index.
|
||||||
|
<2> Sets that up to 1024 soft deletes will be retained.
|
||||||
|
|
||||||
|
[float]
|
||||||
|
[[ccr-getting-started-follower-index]]
|
||||||
|
=== Creating a follower index
|
||||||
|
|
||||||
|
Follower indices are created with the {ref}/ccr-put-follow.html[create follower
|
||||||
|
API]. When you create a follower index, you must reference the
|
||||||
|
<<ccr-getting-started-remote-cluster,remote cluster>> and the
|
||||||
|
<<ccr-getting-started-leader-index,leader index>> that you created in the remote
|
||||||
|
cluster.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT /server-metrics-copy/_ccr/follow
|
||||||
|
{
|
||||||
|
"remote_cluster" : "leader",
|
||||||
|
"leader_index" : "server-metrics"
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[continued]
|
||||||
|
|
||||||
|
//////////////////////////
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
{
|
||||||
|
"follow_index_created" : true,
|
||||||
|
"follow_index_shards_acked" : true,
|
||||||
|
"index_following_started" : true
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// TESTRESPONSE
|
||||||
|
|
||||||
|
//////////////////////////
|
||||||
|
|
||||||
|
Now when you index documents into your leader index, you will see these
|
||||||
|
documents replicated in the follower index. You can
|
||||||
|
inspect the status of replication using the
|
||||||
|
{ref}/ccr-get-follow-stats[get follower stats API].
|
||||||
|
|
||||||
|
//////////////////////////
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
POST /server-metrics-copy/_ccr/pause_follow
|
||||||
|
|
||||||
|
POST /server-metrics-copy/_close
|
||||||
|
|
||||||
|
POST /server-metrics-copy/_ccr/unfollow
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[continued]
|
||||||
|
|
||||||
|
//////////////////////////
|
||||||
|
|
||||||
|
[float]
|
||||||
|
[[ccr-getting-started-auto-follow]]
|
||||||
|
=== Automatically create follower indices
|
||||||
|
|
||||||
|
The auto-follow feature in {ccr} helps for time series use cases where you want
|
||||||
|
to follow new indices that are periodically created in the remote cluster
|
||||||
|
(such as daily Beats indices). Auto-following is configured using the
|
||||||
|
{ref}/ccr-put-auto-follow-pattern.html[create auto-follow pattern API]. With an
|
||||||
|
auto-follow pattern, you reference the
|
||||||
|
<<ccr-getting-started-remote-cluster,remote cluster>> that you connected your
|
||||||
|
local cluster to. You must also specify a collection of patterns that match the
|
||||||
|
indices you want to automatically follow.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT /_ccr/auto_follow/beats
|
||||||
|
{
|
||||||
|
"remote_cluster" : "leader",
|
||||||
|
"leader_index_patterns" :
|
||||||
|
[
|
||||||
|
"metricbeat-*", <1>
|
||||||
|
"packetbeat-*" <2>
|
||||||
|
],
|
||||||
|
"follow_index_pattern" : "{{leader_index}}-copy" <3>
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[continued]
|
||||||
|
<1> Automatically follow new {metricbeat} indices.
|
||||||
|
<2> Automatically follow new {packetbeat} indices.
|
||||||
|
<3> The name of the follower index is derived from the name of the leader index
|
||||||
|
by adding the suffix `-copy` to the name of the leader index.
|
||||||
|
|
||||||
|
//////////////////////////
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
{
|
||||||
|
"acknowledged" : true
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// TESTRESPONSE
|
||||||
|
|
||||||
|
//////////////////////////
|
||||||
|
|
||||||
|
//////////////////////////
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
DELETE /_ccr/auto_follow/beats
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[continued]
|
||||||
|
|
||||||
|
//////////////////////////
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue