Edits from Katie
Signed-off-by: Liz Snyder <elizabsn@amazon.com>
This commit is contained in:
Liz Snyder
parent
e00ba69c08
commit
1e198c0da9
|
|
@ -39,7 +39,7 @@ Options | Description | Type | Required
|
|||
:--- | :--- |:--- |:--- |
|
||||
`leader_alias` | The name of the cross-cluster connection. You define this alias when you [set up a cross-cluster connection]({{site.url}}{{site.baseurl}}/replication-plugin/get-started/#set-up-a-cross-cluster-connection). | `string` | Yes
|
||||
`leader_index` | The index on the leader cluster that you want to replicate. | `string` | Yes
|
||||
`use_roles` | The roles to use for all subsequent backend replication tasks between the indices. Specify a `leader_cluster_role` and `follower_cluster_role`. See [Map the leader and follower cluster roles]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles). | `string` | If security plugin is enabled
|
||||
`use_roles` | The roles to use for all subsequent backend replication tasks between the indexes. Specify a `leader_cluster_role` and `follower_cluster_role`. See [Map the leader and follower cluster roles]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles). | `string` | If security plugin is enabled
|
||||
|
||||
#### Sample response
|
||||
|
||||
|
|
@ -144,13 +144,13 @@ GET /_plugins/_replication/<follower-index>/_status
|
|||
```
|
||||
To include shard replication details in the response, add the `&verbose=true` parameter.
|
||||
|
||||
The leader and follower checkpoint values begin as negative integers and reflect the shard count (-1 for one shard, -5 for five shards, and so on). The values increment toward positive integers with each change that you make. For example, when you make a change on the leader index, the `leader_checkpoint` becomes `0`. The `follower_checkpoint` is initially still `-1` until the follower index pulls the change from the leader, at which point it increments to `0`. If the values are the same, it means the indices are fully synced.
|
||||
The leader and follower checkpoint values begin as negative integers and reflect the shard count (-1 for one shard, -5 for five shards, and so on). The values increment toward positive integers with each change that you make. For example, when you make a change on the leader index, the `leader_checkpoint` becomes `0`. The `follower_checkpoint` is initially still `-1` until the follower index pulls the change from the leader, at which point it increments to `0`. If the values are the same, it means the indexes are fully synced.
|
||||
|
||||
## Get leader cluster stats
|
||||
Introduced 1.1
|
||||
{: .label .label-purple }
|
||||
|
||||
Gets information about replicated leader indices on a specified cluster.
|
||||
Gets information about replicated leader indexes on a specified cluster.
|
||||
|
||||
#### Request
|
||||
|
||||
|
|
@ -197,7 +197,7 @@ GET /_plugins/_replication/leader_stats
|
|||
Introduced 1.1
|
||||
{: .label .label-purple }
|
||||
|
||||
Gets information about follower (syncing) indices on a specified cluster.
|
||||
Gets information about follower (syncing) indexes on a specified cluster.
|
||||
|
||||
#### Request
|
||||
|
||||
|
|
@ -318,7 +318,7 @@ PUT /_plugins/_replication/<follower-index>/_update
|
|||
Introduced 1.1
|
||||
{: .label .label-purple }
|
||||
|
||||
Automatically starts replication on indices matching a specified pattern. If a new index on the leader cluster matches the pattern, OpenSearch automatically creates a follower index and begins replication. You can also use this API to update existing replication rules.
|
||||
Automatically starts replication on indexes matching a specified pattern. If a new index on the leader cluster matches the pattern, OpenSearch automatically creates a follower index and begins replication. You can also use this API to update existing replication rules.
|
||||
|
||||
Send this request to the follower cluster.
|
||||
|
||||
|
|
@ -346,8 +346,8 @@ Options | Description | Type | Required
|
|||
:--- | :--- |:--- |:--- |
|
||||
`leader_alias` | The name of the cross-cluster connection. You define this alias when you [set up a cross-cluster connection]({{site.url}}{{site.baseurl}}/replication-plugin/get-started/#set-up-a-cross-cluster-connection). | `string` | Yes
|
||||
`name` | A name for the auto-follow pattern. | `string` | Yes
|
||||
`pattern` | An array of index patterns to match against indices in the specified leader cluster. Supports wildcard characters. For example, `leader-*`. | `string` | Yes
|
||||
`use_roles` | The roles to use for all subsequent backend replication tasks between the indices. Specify a `leader_cluster_role` and `follower_cluster_role`. See [Map the leader and follower cluster roles]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles). | `string` | If security plugin is enabled
|
||||
`pattern` | An array of index patterns to match against indexes in the specified leader cluster. Supports wildcard characters. For example, `leader-*`. | `string` | Yes
|
||||
`use_roles` | The roles to use for all subsequent backend replication tasks between the indexes. Specify a `leader_cluster_role` and `follower_cluster_role`. See [Map the leader and follower cluster roles]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles). | `string` | If security plugin is enabled
|
||||
|
||||
#### Sample response
|
||||
|
||||
|
|
@ -361,7 +361,7 @@ Options | Description | Type | Required
|
|||
Introduced 1.1
|
||||
{: .label .label-purple }
|
||||
|
||||
Deletes the specified replication rule. This operation prevents any new indices from being replicated but does not stop existing replication that the rule has already initiated. Replicated indices remain read-only until you stop replication.
|
||||
Deletes the specified replication rule. This operation prevents any new indexes from being replicated but does not stop existing replication that the rule has already initiated. Replicated indexes remain read-only until you stop replication.
|
||||
|
||||
Send this request to the follower cluster.
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ has_children: false
|
|||
|
||||
# Auto-follow for cross-cluster replication
|
||||
|
||||
Auto-follow lets you automatically replicate indices created on the leader cluster based on matching patterns. When you create an index on the leader cluster with a name that matches a specified pattern (for example, `index-01*`), a corresponding follower index is automatically created on the follower cluster.
|
||||
Auto-follow lets you automatically replicate indexes created on the leader cluster based on matching patterns. When you create an index on the leader cluster with a name that matches a specified pattern (for example, `index-01*`), a corresponding follower index is automatically created on the follower cluster.
|
||||
|
||||
You can configure multiple replication rules for a single cluster. The patterns currently only support wildcard matching.
|
||||
|
||||
|
|
@ -18,11 +18,11 @@ You need to [set up a cross-cluster connection]({{site.url}}{{site.baseurl}}/rep
|
|||
|
||||
## Permissions
|
||||
|
||||
If the security plugin is enabled, non-admin users need to be mapped to the appropriate permissions in order to perform replication actions. For index and cluster-level permissions requirements, see [Cross-cluster replication permissions]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/).
|
||||
If the security plugin is enabled, make sure that non-admin users are mapped to the appropriate permissions so they can perform replication actions. For index and cluster-level permissions requirements, see [Cross-cluster replication permissions]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/).
|
||||
|
||||
## Get started with auto-follow
|
||||
|
||||
Replication rules are a collection of patterns that you create against a single remote cluster. When you create a replication rule, it automatically starts replicating any *new* indices that match the pattern, but does not replicate matching indices that were previously created.
|
||||
Replication rules are a collection of patterns that you create against a single remote cluster. When you create a replication rule, it automatically starts replicating any *new* indexes that match the pattern, but does not replicate matching indexes that were previously created.
|
||||
|
||||
Create a replication rule on the follower cluster:
|
||||
|
||||
|
|
@ -63,7 +63,7 @@ yellow open movies-0001 kHOxYYHxRMeszLjTD9rvSQ 1 1 0
|
|||
|
||||
## Retrieve replication rules
|
||||
|
||||
To retrieve a list of existing replication rules configured on a cluster, send the following request:
|
||||
To retrieve a list of existing replication rules that are configured on a cluster, send the following request:
|
||||
|
||||
```bash
|
||||
curl -XGET -u 'admin:admin' -k 'https://localhost:9200/_plugins/_replication/autofollow_stats'
|
||||
|
|
@ -102,5 +102,5 @@ curl -XDELETE -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://l
|
|||
}'
|
||||
```
|
||||
|
||||
OpenSearch stops replicating *new* indices that match the pattern, but existing indices that the rule previously created remain read-only and continue to replicate. If you need to stop existing replication activity and open the indices up for writes, use the [stop replication API operation]({{site.url}}{{site.baseurl}}/replication-plugin/api/#stop-replication).
|
||||
OpenSearch stops replicating *new* indexes that match the pattern, but existing indexes that the rule previously created remain read-only and continue to replicate. If you need to stop existing replication activity and open the indexes up for writes, use the [stop replication API operation]({{site.url}}{{site.baseurl}}/replication-plugin/api/#stop-replication).
|
||||
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ nav_order: 10
|
|||
|
||||
# Get started with cross-cluster replication
|
||||
|
||||
With cross-cluster replication, you index data to a leader index, and OpenSearch replicates that data to one or more read-only follower indices. All subsequent operations on the leader are replicated on the follower, such as creating, updating, or deleting documents.
|
||||
With cross-cluster replication, you index data to a leader index, and OpenSearch replicates that data to one or more read-only follower indexes. All subsequent operations on the leader are replicated on the follower, such as creating, updating, or deleting documents.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
|
|
@ -22,9 +22,9 @@ Cross-cluster replication has the following prerequisites:
|
|||
|
||||
Make sure the security plugin is either enabled on both clusters or disabled on both clusters. If you disabled the security plugin, you can skip this section. However, we strongly recommend enabling the security plugin in production scenarios.
|
||||
|
||||
If the security plugin is enabled, non-admin users need to be mapped to the appropriate permissions in order to perform replication actions. For index and cluster-level permissions requirements, see [Cross-cluster replication permissions]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/).
|
||||
If the security plugin is enabled, make sure that non-admin users are mapped to the appropriate permissions so they can perform replication actions. For index and cluster-level permissions requirements, see [Cross-cluster replication permissions]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/).
|
||||
|
||||
In addition, verify and add the Distinguished names (DNs) of each follower cluster node on the leader cluster to allow connections from the followers to the leader.
|
||||
In addition, verify and add the distinguished names (DNs) of each follower cluster node on the leader cluster to allow connections from the followers to the leader.
|
||||
|
||||
First, get the node's DN from each follower cluster:
|
||||
|
||||
|
|
@ -43,7 +43,7 @@ curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_plugins/_security/api/ss
|
|||
}
|
||||
```
|
||||
|
||||
Then verify that it's part of the leader cluster configuration in `opensearch.yml`, otherwise add it under the following setting:
|
||||
Then verify that it's part of the leader cluster configuration in `opensearch.yml`. Otherwise, add it under the following setting:
|
||||
|
||||
```yaml
|
||||
plugins.security.nodes_dn:
|
||||
|
|
@ -184,7 +184,7 @@ curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://loca
|
|||
If the security plugin is disabled, omit the `use_roles` parameter. If it's enabled, however, you must specify the leader and follower cluster roles that OpenSearch will use to authenticate the request. This example uses `all_access` for simplicity, but we recommend creating a replication user on each cluster and [mapping it accordingly]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles).
|
||||
{: .tip }
|
||||
|
||||
This command creates an identical read-only index named `follower-01` on the follower cluster that continuously stays updated with changes to the `leader-01` index on the leader cluster. Starting replication creates a follower index from scratch; you can't convert an existing index to a follower index.
|
||||
This command creates an identical read-only index named `follower-01` on the follower cluster that continuously stays updated with changes to the `leader-01` index on the leader cluster. Starting replication creates a follower index from scratch -- you can't convert an existing index to a follower index.
|
||||
|
||||
## Confirm replication
|
||||
|
||||
|
|
@ -207,9 +207,9 @@ curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/fol
|
|||
}
|
||||
```
|
||||
|
||||
Possible statuses are `SYNCING`, `BOOTSTRAPING`, `PAUSED`, and `REPLICATION NOT IN PROGRESS`.
|
||||
Possible statuses are `SYNCING`, `BOOTSTRAPPING`, `PAUSED`, and `REPLICATION NOT IN PROGRESS`.
|
||||
|
||||
The leader and follower checkpoint values begin as negative numbers and reflect the shard count (-1 for one shard, -5 for five shards, and so on). The values increment with each change and illustrate how many updates the follower is behind the leader. If the indices are fully synced, the values are the same.
|
||||
The leader and follower checkpoint values begin as negative numbers and reflect the shard count (-1 for one shard, -5 for five shards, and so on). The values increment with each change and illustrate how many updates the follower is behind the leader. If the indexes are fully synced, the values are the same.
|
||||
|
||||
To confirm that replication is actually happening, add a document to the leader index:
|
||||
|
||||
|
|
@ -244,7 +244,7 @@ You can temporarily pause replication of an index if you need to remediate issue
|
|||
curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_pause?pretty' -d '{}'
|
||||
```
|
||||
|
||||
To confirm replication is paused, get the status:
|
||||
To confirm that replication is paused, get the status:
|
||||
|
||||
```bash
|
||||
curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_status?pretty'
|
||||
|
|
@ -270,7 +270,7 @@ Note that you can't resume replication after it's been paused for more than 12 h
|
|||
|
||||
## Stop replication
|
||||
|
||||
Terminate replication of a specified index from the follower cluster:
|
||||
When you no longer need to replicate an index, terminate replication from the follower cluster:
|
||||
|
||||
```bash
|
||||
curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_stop?pretty' -d '{}'
|
||||
|
|
|
|||
|
|
@ -9,14 +9,14 @@ redirect_from:
|
|||
|
||||
# Cross-cluster replication
|
||||
|
||||
The cross-cluster replication plugin lets you replicate indices, mappings, and metadata from one OpenSearch cluster to another. Cross-cluster replication has the following benefits:
|
||||
- By replicating your indices, you ensure that you can continue to handle search requests in the event of an outage.
|
||||
The cross-cluster replication plugin lets you replicate indexes, mappings, and metadata from one OpenSearch cluster to another. Cross-cluster replication has the following benefits:
|
||||
- By replicating your indexes, you ensure that you can continue to handle search requests in the event of an outage.
|
||||
- Replicating data across geographically distant data centers minimizes the distance between the data and the application server, reducing expensive latencies.
|
||||
- You can replicate data from multiple smaller clusters to a centralized reporting cluster, which is useful when it's inefficient to query across a large network.
|
||||
|
||||
Replication follows an active-passive model where the follower index (where the data is replicated) pulls data from the leader (remote) index.
|
||||
|
||||
The replication plugin supports replication of indices using wildcard pattern matching and provides commands to pause, resume, and stop replication. Once replication starts on an index, it initiates persistent background tasks on all primary shards on the follower cluster, which continuously poll corresponding shards from the leader cluster for updates.
|
||||
The replication plugin supports replication of indexes using wildcard pattern matching and provides commands to pause, resume, and stop replication. Once replication starts on an index, it initiates persistent background tasks on all primary shards on the follower cluster, which continuously poll corresponding shards from the leader cluster for updates.
|
||||
|
||||
You can use the replication plugin with the security plugin to encrypt cross-cluster traffic with node-to-node encryption and control access to replication activities.
|
||||
|
||||
|
|
|
|||
|
|
@ -25,12 +25,12 @@ These settings manage the resources consumed by remote recoveries. We don’t re
|
|||
|
||||
Setting | Default | Description
|
||||
:--- | :--- | :---
|
||||
`plugins.replication.follower.index.recovery.chunk_size` | 10MB | The chunk size requested by the follower cluster during file transfer. Specify the chunk size as a value and unit, for example, 10MB, 5KB. See [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/).
|
||||
`plugins.replication.follower.index.recovery.chunk_size` | 10 MB | The chunk size requested by the follower cluster during file transfer. Specify the chunk size as a value and unit, for example, 10 MB, 5 KB. See [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/).
|
||||
`plugins.replication.follower.index.recovery.max_concurrent_file_chunks` | 4 | The number of file chunk requests that can be sent in parallel for each recovery.
|
||||
`plugins.replication.follower.index.ops_batch_size` | 5000 | The number of operations that can be fetched at a time during the syncing phase of replication.
|
||||
`plugins.replication.follower.concurrent_readers_per_shard` | 2 | The number of concurrent requests from the follower cluster per shard during the syncing phase of replication.
|
||||
`plugins.replication.autofollow.fetch_poll_interval` | 30s | How often auto-follow tasks poll the leader cluster for new matching indices.
|
||||
`plugins.replication.autofollow.fetch_poll_interval` | 30s | How often auto-follow tasks poll the leader cluster for new matching indexes.
|
||||
`plugins.replication.follower.metadata_sync_interval` | 60s | How often the follower cluster polls the leader cluster for updated index metadata.
|
||||
`plugins.replication.translog.retention_lease.pruning.enabled` | true | If enabled, prunes the translog based on retention leases on the leader index.
|
||||
`plugins.replication.translog.retention_size` | 512MB | Controls the size of the translog on the leader index.
|
||||
`plugins.replication.translog.retention_size` | 512 MB | Controls the size of the translog on the leader index.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue