* Applying some initial changes. * Updating intro and screenshots. * Removing unnecessary links, streamlining content, and adding GIF. * Adding what's next section. * Removing what's next. * Minor edits. * Apply suggestions from code review Co-authored-by: debadair <debadair@elastic.co> * Incorporating review feedback. * Moving CCR user privileges to another page, plus more edits. * Apply suggestions from code review Co-authored-by: debadair <debadair@elastic.co> * Incorporating more review feedback. * Adding TESTSETUP to fix build errors. * Update docs/reference/ccr/getting-started.asciidoc Co-authored-by: debadair <debadair@elastic.co> * Swapping GIF for mp4 hosted on web team CMS. * Removing GIF in favor of mp4. Co-authored-by: debadair <debadair@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: debadair <debadair@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
|
@ -1,90 +1,97 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[testenv="platinum"]
|
[testenv="platinum"]
|
||||||
[[ccr-getting-started]]
|
[[ccr-getting-started]]
|
||||||
=== Set up {ccr}
|
=== Tutorial: Set up {ccr}
|
||||||
|
++++
|
||||||
|
<titleabbrev>Set up {ccr}</titleabbrev>
|
||||||
|
++++
|
||||||
|
|
||||||
|
////
|
||||||
|
[source,console]
|
||||||
|
----
|
||||||
|
PUT /server-metrics
|
||||||
|
{
|
||||||
|
"settings" : {
|
||||||
|
"index" : {
|
||||||
|
"number_of_shards" : 1,
|
||||||
|
"number_of_replicas" : 0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"mappings" : {
|
||||||
|
"properties" : {
|
||||||
|
"@timestamp" : {
|
||||||
|
"type" : "date"
|
||||||
|
},
|
||||||
|
"accept" : {
|
||||||
|
"type" : "long"
|
||||||
|
},
|
||||||
|
"deny" : {
|
||||||
|
"type" : "long"
|
||||||
|
},
|
||||||
|
"host" : {
|
||||||
|
"type" : "keyword"
|
||||||
|
},
|
||||||
|
"response" : {
|
||||||
|
"type" : "float"
|
||||||
|
},
|
||||||
|
"service" : {
|
||||||
|
"type" : "keyword"
|
||||||
|
},
|
||||||
|
"total" : {
|
||||||
|
"type" : "long"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
----
|
||||||
|
// TESTSETUP
|
||||||
|
////
|
||||||
|
|
||||||
|
Use this guide to set up {ccr} (CCR) between clusters in two
|
||||||
|
datacenters. Replicating your data across datacenters provides several benefits:
|
||||||
|
|
||||||
|
* Brings data closer to your users or application server to reduce latency and
|
||||||
|
response time
|
||||||
|
* Provides your mission-critical applications with the tolerance to withstand datacenter or region outages
|
||||||
|
|
||||||
|
In this guide, you'll learn how to:
|
||||||
|
|
||||||
|
* Configure a <<modules-remote-clusters,remote cluster>> with a leader index
|
||||||
|
* Create a follower index on a local cluster
|
||||||
|
* Create an auto-follow pattern to automatically follow time series indices
|
||||||
|
that are periodically created in a remote cluster
|
||||||
|
|
||||||
You can manually create follower indices to replicate specific indices on a
|
You can manually create follower indices to replicate specific indices on a
|
||||||
remote cluster, or configure auto-follow patterns to automatically create
|
remote cluster, or configure auto-follow patterns to replicate rolling time series indices.
|
||||||
follower indices for new time series.
|
|
||||||
|
|
||||||
After the follower index is created, the
|
video::https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt994089f5e841ad69/5f6265de6f40ab4648b5cf9b/ccr-setup-video-edited.mp4[width=700, height=500, options="autoplay,loop"]
|
||||||
<<ccr-remote-recovery, remote recovery>> process copies all of the Lucene
|
|
||||||
segment files from the remote cluster to the local cluster.
|
|
||||||
|
|
||||||
To set up {ccr}:
|
|
||||||
|
|
||||||
. <<ccr-getting-started-remote-cluster,Connect a local cluster to a remote cluster>>
|
|
||||||
. <<ccr-getting-started-leader-index,Identify the index (or time series indices) you want to replicate on the remote cluster>>
|
|
||||||
. <<ccr-enable-soft-deletes,Enable soft deletes on the leader index>>
|
|
||||||
. Manually create a follower index or create an auto-follow pattern:
|
|
||||||
* To replicate the leader index, <<ccr-getting-started-follower-index,manually create a follower index>>
|
|
||||||
* To automatically follow time series indices, <<ccr-getting-started-auto-follow,create an auto-follow pattern>>
|
|
||||||
|
|
||||||
[[ccr-getting-started-prerequisites]]
|
[[ccr-getting-started-prerequisites]]
|
||||||
==== Prerequisites
|
==== Prerequisites
|
||||||
If the Elastic {security-features} are enabled in your local and remote
|
To complete this tutorial, you need:
|
||||||
clusters, you need a user with appropriate authority to complete the steps
|
|
||||||
in this tutorial.
|
|
||||||
|
|
||||||
By default, you can complete the following steps as the built-in
|
* A license on both clusters that includes {ccr}. {kibana-ref}/managing-licenses.html[Activate a free 30-day trial].
|
||||||
`elastic` user. However, you must <<get-started-built-in-users,set a password>>
|
* The `read_ccr` cluster privilege and `monitor` and `read` privileges
|
||||||
for this user before proceeding.
|
for the leader index on the remote cluster. <<stack-management-ccr-remote,Configure remote cluster privileges>>.
|
||||||
|
* The `manage_ccr` cluster privilege and `monitor`, `read`, `write` and
|
||||||
WARNING: If you are performing these steps in a production environment, do
|
`manage_follow_index` privileges to configure remote clusters and follower
|
||||||
not use the `elastic` user.
|
indices on the local cluster. <<stack-management-ccr-local,Configure local cluster privileges>>.
|
||||||
|
* An index on the remote cluster that contains the data you want to replicate.
|
||||||
Alternatively, you can assign the appropriate privileges to a user ID of your
|
This tutorial uses the sample eCommerce orders data set.
|
||||||
choice. On the remote cluster that contains the leader index, a user must have
|
{kibana-ref}/get-started.html#gs-get-data-into-kibana[Load sample data].
|
||||||
the `read_ccr` cluster privilege and `monitor` and `read` privileges on the
|
|
||||||
leader index.
|
|
||||||
|
|
||||||
[source,yml]
|
|
||||||
--------------------------------------------------
|
|
||||||
ccr_user:
|
|
||||||
cluster:
|
|
||||||
- read_ccr
|
|
||||||
indices:
|
|
||||||
- names: [ 'leader-index' ]
|
|
||||||
privileges:
|
|
||||||
- monitor
|
|
||||||
- read
|
|
||||||
--------------------------------------------------
|
|
||||||
|
|
||||||
On the local cluster that contains the follower index, the same user will need
|
|
||||||
the `manage_ccr` cluster privilege and `monitor`, `read`, `write` and
|
|
||||||
`manage_follow_index` privileges on the follower index.
|
|
||||||
|
|
||||||
[source,yml]
|
|
||||||
--------------------------------------------------
|
|
||||||
ccr_user:
|
|
||||||
cluster:
|
|
||||||
- manage_ccr
|
|
||||||
indices:
|
|
||||||
- names: [ 'follower-index' ]
|
|
||||||
privileges:
|
|
||||||
- monitor
|
|
||||||
- read
|
|
||||||
- write
|
|
||||||
- manage_follow_index
|
|
||||||
--------------------------------------------------
|
|
||||||
|
|
||||||
If you are managing
|
|
||||||
<<ccr-getting-started-remote-cluster,connecting to the remote cluster>> using
|
|
||||||
the cluster update settings API, you will also need a user with the `all`
|
|
||||||
cluster privilege.
|
|
||||||
|
|
||||||
[[ccr-getting-started-remote-cluster]]
|
[[ccr-getting-started-remote-cluster]]
|
||||||
==== Connect to a remote cluster
|
==== Connect to a remote cluster
|
||||||
Connect your local cluster to a
|
To replicate an index on a remote cluster (Cluster A) to a local cluster (Cluster B), you configure Cluster A as a remote on Cluster B.
|
||||||
<<modules-remote-clusters,remote cluster>> to begin using cross-cluster
|
|
||||||
replication.
|
|
||||||
|
|
||||||
To configure a {kibana-ref}/working-remote-clusters.html[remote cluster],
|
image::images/ccr-tutorial-clusters.png[ClusterA contains the leader index and ClusterB contains the follower index]
|
||||||
access {kib} and go to
|
|
||||||
*Management > Stack Management*. In the side navigation, select
|
|
||||||
*Remote Clusters*.
|
|
||||||
|
|
||||||
Add a remote cluster by specifying the IP address or host name, followed by the
|
To configure a remote cluster from Stack Management in {kib}:
|
||||||
transport port of the remote cluster.
|
|
||||||
|
. Select *Remote Clusters* from the side navigation.
|
||||||
|
. Specify the IP address or host name of the remote cluster (ClusterB),
|
||||||
|
followed by the transport port of the remote cluster (defaults to `9300`). For
|
||||||
|
example, `192.168.1.1:9300`.
|
||||||
|
|
||||||
[role="screenshot"]
|
[role="screenshot"]
|
||||||
image::images/ccr-add-remote-cluster.png["The Add remote clusters page in {kib}"]
|
image::images/ccr-add-remote-cluster.png["The Add remote clusters page in {kib}"]
|
||||||
|
@ -153,96 +160,39 @@ remote cluster.
|
||||||
connected to.
|
connected to.
|
||||||
====
|
====
|
||||||
|
|
||||||
[[ccr-getting-started-leader-index]]
|
|
||||||
==== Create a leader index
|
|
||||||
To create a leader index, access {kib} on your _remote_ cluster and go to
|
|
||||||
*Management > Dev Tools*.
|
|
||||||
|
|
||||||
Copy the following example into the Console to create a leader index named
|
|
||||||
`server-metrics` in your remote cluster:
|
|
||||||
|
|
||||||
[%collapsible]
|
|
||||||
.Leader index example
|
|
||||||
====
|
|
||||||
[source,console]
|
|
||||||
--------------------------------------------------
|
|
||||||
PUT /server-metrics
|
|
||||||
{
|
|
||||||
"settings" : {
|
|
||||||
"index" : {
|
|
||||||
"number_of_shards" : 1,
|
|
||||||
"number_of_replicas" : 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"mappings" : {
|
|
||||||
"properties" : {
|
|
||||||
"@timestamp" : {
|
|
||||||
"type" : "date"
|
|
||||||
},
|
|
||||||
"accept" : {
|
|
||||||
"type" : "long"
|
|
||||||
},
|
|
||||||
"deny" : {
|
|
||||||
"type" : "long"
|
|
||||||
},
|
|
||||||
"host" : {
|
|
||||||
"type" : "keyword"
|
|
||||||
},
|
|
||||||
"response" : {
|
|
||||||
"type" : "float"
|
|
||||||
},
|
|
||||||
"service" : {
|
|
||||||
"type" : "keyword"
|
|
||||||
},
|
|
||||||
"total" : {
|
|
||||||
"type" : "long"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
--------------------------------------------------
|
|
||||||
// TEST[continued]
|
|
||||||
====
|
|
||||||
|
|
||||||
[[ccr-enable-soft-deletes]]
|
[[ccr-enable-soft-deletes]]
|
||||||
==== Enable soft deletes on leader indices
|
==== Enable soft deletes on leader indices
|
||||||
<<ccr-leader-requirements,Soft deletes>> must be enabled for indices that you want to
|
To follow an index, it must have been created with
|
||||||
use as leader indices. Soft deletes are enabled by default on new indices
|
<<ccr-leader-requirements,soft deletes>> enabled. If the index doesn’t have
|
||||||
created on or after {es} 7.0.0, so
|
soft deletes enabled, you must reindex it and use the new index as the leader
|
||||||
*no further action is required if your cluster is running {es} 7.0.0 or later*.
|
index. Soft deletes are enabled by default on new indices
|
||||||
|
created with {es} 7.0.0 and later.
|
||||||
include::{es-ref-dir}/ccr/index.asciidoc[tag=ccr-existing-indices-tag]
|
|
||||||
|
|
||||||
To enable soft deletes on indices created on versions of
|
|
||||||
{es} between 6.5.0 and 7.0.0, set <<ccr-index-soft-deletes,`index.soft_deletes.enabled`>> to `true`.
|
|
||||||
|
|
||||||
[[ccr-getting-started-follower-index]]
|
[[ccr-getting-started-follower-index]]
|
||||||
==== Create a follower index
|
==== Create a follower index to replicate a specific index
|
||||||
When you create a {kibana-ref}/managing-cross-cluster-replication.html#_create_specific_follower_indices[follower index], you
|
When you create a follower index, you reference the remote cluster and the
|
||||||
must reference the
|
leader index in your remote cluster.
|
||||||
<<ccr-getting-started-remote-cluster,remote cluster>> and the
|
|
||||||
<<ccr-getting-started-leader-index,leader index>> that you created in the remote
|
|
||||||
cluster.
|
|
||||||
|
|
||||||
To create a follower index, access {kib} and go to
|
To create a follower index from Stack Management in {kib}:
|
||||||
*Management > Stack Management*. In the side navigation, select
|
|
||||||
*Cross-Cluster Replication* and choose the *Follower Indices* tab.
|
|
||||||
|
|
||||||
. Choose the remote cluster containing the index you want to replicate, which
|
. Select *Cross-Cluster Replication* in the side navigation and choose the
|
||||||
is `leader` if you are following the tutorial.
|
*Follower Indices* tab.
|
||||||
. Enter the name of the leader index, which is `server-metrics` if you are
|
. Choose the cluster (ClusterA) containing the leader index you want to
|
||||||
following the tutorial.
|
replicate.
|
||||||
|
. Enter the name of the leader index, which is
|
||||||
|
`kibana_sample_data_ecommerce` if you are following the tutorial.
|
||||||
|
. Enter a name for your follower index, such as `follower-kibana-sample-data`.
|
||||||
|
|
||||||
image::images/ccr-add-follower-index.png["Adding a follower index named server-metrics in {kib}"]
|
image::images/ccr-add-follower-index.png["Adding a follower index named server-metrics in {kib}"]
|
||||||
|
|
||||||
The follower index is initialized using the
|
{es} initializes the follower using the
|
||||||
<<ccr-remote-recovery, remote recovery>>
|
<<ccr-remote-recovery, remote recovery>>
|
||||||
process, which transfers the existing Lucene segment files from the leader
|
process, which transfers the existing Lucene segment files from the leader
|
||||||
index to the follower index. The index status changes to *Paused*. When the
|
index to the follower index. The index status changes to *Paused*. When the
|
||||||
remote recovery process is complete, the index following begins and the status
|
remote recovery process is complete, the index following begins and the status
|
||||||
changes to *Active*.
|
changes to *Active*.
|
||||||
|
|
||||||
When you index documents into your leader index, the documents are replicated
|
When you index documents into your leader index, {es} replicates the documents
|
||||||
in the follower index.
|
in the follower index.
|
||||||
|
|
||||||
[role="screenshot"]
|
[role="screenshot"]
|
||||||
|
@ -252,9 +202,8 @@ image::images/ccr-follower-index.png["The Cross-Cluster Replication page in {kib
|
||||||
.API example
|
.API example
|
||||||
====
|
====
|
||||||
Use the <<ccr-put-follow,create follower API>> to create follower indices.
|
Use the <<ccr-put-follow,create follower API>> to create follower indices.
|
||||||
When you create a follower index, you must reference the
|
When you create a follower index, you must reference the remote cluster and the
|
||||||
<<ccr-getting-started-remote-cluster,remote cluster>> and the
|
leader index that you created in the
|
||||||
<<ccr-getting-started-leader-index,leader index>> that you created in the
|
|
||||||
remote cluster.
|
remote cluster.
|
||||||
|
|
||||||
When initiating the follower request, the response returns before the
|
When initiating the follower request, the response returns before the
|
||||||
|
@ -304,37 +253,35 @@ POST /server-metrics-follower/_ccr/unfollow
|
||||||
====
|
====
|
||||||
|
|
||||||
[[ccr-getting-started-auto-follow]]
|
[[ccr-getting-started-auto-follow]]
|
||||||
==== Automatically create follower indices
|
==== Create an auto-follow pattern to replicate time series indices
|
||||||
Create <<ccr-auto-follow,auto-follow patterns>> to automatically follow time
|
You use <<ccr-auto-follow,auto-follow patterns>> to automatically create new
|
||||||
series indices that are periodically created in a remote cluster (such as daily
|
followers for rolling time series indices. Whenever the name of a new index on
|
||||||
{beats} indices).
|
the remote cluster matches the auto-follow pattern, a corresponding follower
|
||||||
|
index is added to the local cluster.
|
||||||
|
|
||||||
With an auto-follow pattern, you reference the
|
An auto-follow pattern specifies the remote cluster you want to replicate from,
|
||||||
<<ccr-getting-started-remote-cluster,remote cluster>> connected to your
|
and one or more index patterns that specify the rolling time series indices you
|
||||||
local cluster. You must also specify a collection of patterns that match the
|
want to replicate.
|
||||||
indices you want to automatically follow.
|
|
||||||
|
|
||||||
// tag::ccr-create-auto-follow-pattern-tag[]
|
// tag::ccr-create-auto-follow-pattern-tag[]
|
||||||
To create follower indices from an {kibana-ref}/managing-cross-cluster-replication.html#_create_follower_indices_from_an_auto_follow_pattern[auto-follow pattern],
|
To create an auto-follow pattern from Stack Management in {kib}:
|
||||||
access {kib} on your remote cluster and go to
|
|
||||||
*Management > Stack Management*. In the side navigation, select
|
|
||||||
*Cross Cluster Replication* and choose the *Auto-follow patterns* tab.
|
|
||||||
|
|
||||||
[role="screenshot"]
|
. Select *Cross Cluster Replication* in the side navigation and choose the
|
||||||
image::images/auto-follow-patterns.png["The Auto-follow patterns page in {kib}"]
|
*Auto-follow patterns* tab.
|
||||||
|
. Enter a name for the auto-follow pattern, such as `beats`.
|
||||||
* Enter a name for the auto-follow pattern. For this tutorial, enter `beats`
|
. Choose the remote cluster that contains the index you want to replicate,
|
||||||
as the name.
|
which in the example scenario is Cluster A.
|
||||||
* Choose the remote cluster containing the index you want to replicate, which
|
. Enter one or more index patterns that identify the indices you want to
|
||||||
is `leader` if you are following the tutorial.
|
replicate from the remote cluster. For example, enter
|
||||||
* Enter one or more index patterns that identify the indices you want to
|
`metricbeat-* packetbeat-*` to automatically create followers for {metricbeat} and {packetbeat} indices.
|
||||||
replicate from the remote cluster. For this tutorial, enter
|
. Enter *follower-* as the prefix to apply to the names of the follower indices so
|
||||||
`metricbeat-*,packetbeat-*` as the index pattern.
|
|
||||||
* Enter *copy-* as the prefix to apply to the names of the follower indices so
|
|
||||||
you can more easily identify replicated indices.
|
you can more easily identify replicated indices.
|
||||||
|
|
||||||
As new indices matching these patterns are
|
As new indices matching these patterns are
|
||||||
created, they are replicated to the follower indices.
|
created on the remote, {es} automatically replicates them to local follower indices.
|
||||||
|
|
||||||
|
[role="screenshot"]
|
||||||
|
image::images/auto-follow-patterns.png["The Auto-follow patterns page in {kib}"]
|
||||||
|
|
||||||
// end::ccr-create-auto-follow-pattern-tag[]
|
// end::ccr-create-auto-follow-pattern-tag[]
|
||||||
|
|
||||||
|
|
After Width: | Height: | Size: 357 KiB |
Before Width: | Height: | Size: 162 KiB After Width: | Height: | Size: 159 KiB |
Before Width: | Height: | Size: 273 KiB After Width: | Height: | Size: 280 KiB |
Before Width: | Height: | Size: 279 KiB After Width: | Height: | Size: 279 KiB |
Before Width: | Height: | Size: 174 KiB After Width: | Height: | Size: 189 KiB |
After Width: | Height: | Size: 59 KiB |
|
@ -5,6 +5,8 @@ include::built-in-roles.asciidoc[]
|
||||||
|
|
||||||
include::managing-roles.asciidoc[]
|
include::managing-roles.asciidoc[]
|
||||||
|
|
||||||
|
include::stack-management.asciidoc[]
|
||||||
|
|
||||||
include::privileges.asciidoc[]
|
include::privileges.asciidoc[]
|
||||||
|
|
||||||
include::document-level-security.asciidoc[]
|
include::document-level-security.asciidoc[]
|
||||||
|
|
|
@ -0,0 +1,50 @@
|
||||||
|
[role="xpack"]
|
||||||
|
[[stack-management]]
|
||||||
|
=== Granting access to Stack Management features
|
||||||
|
You <<defining-roles,define roles>> and set user privileges at different levels
|
||||||
|
to grant access to each of the Elastic Stack features.
|
||||||
|
|
||||||
|
[[stack-management-ccr]]
|
||||||
|
==== {ccr-cap}
|
||||||
|
The {ccr} user requires different cluster and index privileges on the remote
|
||||||
|
cluster and local cluster.
|
||||||
|
|
||||||
|
[[stack-management-ccr-remote]]
|
||||||
|
On the remote cluster that contains the leader index, the {ccr} user requires
|
||||||
|
`read_ccr` cluster privilege and `monitor` and `read` privileges on the
|
||||||
|
leader index.
|
||||||
|
|
||||||
|
[source,yml]
|
||||||
|
--------------------------------------------------
|
||||||
|
ccr_user:
|
||||||
|
cluster:
|
||||||
|
- read_ccr
|
||||||
|
indices:
|
||||||
|
- names: [ 'leader-index' ]
|
||||||
|
privileges:
|
||||||
|
- monitor
|
||||||
|
- read
|
||||||
|
--------------------------------------------------
|
||||||
|
|
||||||
|
[[stack-management-ccr-local]]
|
||||||
|
On the local cluster that contains the follower index, the {ccr} user requires the `manage_ccr` cluster privilege and `monitor`, `read`, `write` and
|
||||||
|
`manage_follow_index` privileges on the follower index.
|
||||||
|
|
||||||
|
[source,yml]
|
||||||
|
--------------------------------------------------
|
||||||
|
ccr_user:
|
||||||
|
cluster:
|
||||||
|
- manage_ccr
|
||||||
|
indices:
|
||||||
|
- names: [ 'follower-index' ]
|
||||||
|
privileges:
|
||||||
|
- monitor
|
||||||
|
- read
|
||||||
|
- write
|
||||||
|
- manage_follow_index
|
||||||
|
--------------------------------------------------
|
||||||
|
|
||||||
|
If you are managing
|
||||||
|
<<ccr-getting-started-remote-cluster,connecting to the remote cluster>> using
|
||||||
|
the cluster update settings API, you will also need a user with the `all`
|
||||||
|
cluster privilege.
|