iSharkFly-Docs
/
opensearch-docs-cn
mirror of https://github.com/iSharkFly-Docs/opensearch-docs-cn
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Incorporate feedback and random fixes

This commit is contained in:
Liz Snyder 2021-10-01 11:43:37 -07:00
parent 8d3ae42f6a
commit cdff560cf6
6 changed files with 79 additions and 61 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
_replication-plugin/api.md
View File

@ -16,7 +16,7 @@ Use these replication operations to programmatically manage cross-cluster replic
Introduced 1.1
{: .label .label-purple }
Initiate replication of an index from the leader cluster to the follower cluster. Run this operation on the follower cluster.
Initiate replication of an index from the leader cluster to the follower cluster. Send this request to the follower cluster.
#### Request
@ -37,7 +37,7 @@ Specify the following options:
Options | Description | Type | Required
:--- | :--- |:--- |:--- |
`leader_alias` | The name of the leader cluster. | `string` | Yes
`leader_alias` | The name of the leader cluster. This alias is the same as the remote cluster name used to set up a cross-cluster connection. | `string` | Yes
`leader_index` | The index on the leader cluster that you want to replicate. | `string` | Yes
`use_roles` | The roles to use for all subsequent backend replication tasks between the indices. Specify a `leader_cluster_role` and `follower_cluster_role`. See [Map the leader and follower cluster roles]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles). | `string` | If security plugin is enabled
@ -53,12 +53,13 @@ Options | Description | Type | Required
Introduced 1.1
{: .label .label-purple }
Terminates replication and converts the follower index to a standard index.
Terminates replication and converts the follower index to a standard index. Send this request to the follower cluster.
#### Request
```json
POST /_plugins/_replication/<follower-index>/_stop
{}
```
#### Sample response
@ -73,12 +74,13 @@ POST /_plugins/_replication/<follower-index>/_stop
Introduced 1.1
{: .label .label-purple }
Pauses replication of the leader index. If you don't resume replication after 12 hours, it stops completely and the follower index is converted to a standard index.
Pauses replication of the leader index. Send this request to the follower cluster.
#### Request
```json
PUT /_plugins/_replication/<follower-index>/_pause
POST /_plugins/_replication/<follower-index>/_pause
{}
```
#### Sample response
@ -93,12 +95,13 @@ PUT /_plugins/_replication/<follower-index>/_pause
Introduced 1.1
{: .label .label-purple }
Resumes replication of the leader index.
Resumes replication of the leader index. Send this request to the follower cluster.
#### Request
```json
PUT /_plugins/_replication/<follower-index>/_resume
POST /_plugins/_replication/<follower-index>/_resume
{}
```
#### Sample response
@ -113,7 +116,7 @@ PUT /_plugins/_replication/<follower-index>/_resume
Introduced 1.1
{: .label .label-purple }
Gets the status of index replication. You can use this API to measure replication lag. Run this command from the leader cluster.
Gets the status of index replication. Possible statuses are `SYNCING`, `BOOTSTRAPING`, `PAUSED`, and `REPLICATION NOT IN PROGRESS`. Use the syncing details to measure replication lag. Send this request to the follower cluster.
#### Request
@ -125,20 +128,20 @@ GET /_plugins/_replication/<follower-index>/_status
```json
{
"status":"SYNCING",
"reason":"User initiated",
"remote_cluster":"remote-cluster",
"leader_index":"leader-01",
"follower_index":"follower-01",
"syncing_details":{
"remote_checkpoint": 19,
"local_checkpoint": 19,
"seq_no": 20
}
"status" : "SYNCING",
"reason" : "User initiated",
"leader_alias" : "leader-cluster",
"leader_index" : "leader-01",
"follower_index" : "follower-01",
"syncing_details" : {
"leader_checkpoint" : 19,
"follower_checkpoint" : 19,
"seq_no" : 0
}
}
```
To include shard replication details in the response, add `&verbose=true`.
To include shard replication details in the response, add the `&verbose=true` parameter.
## Update settings
Introduced 1.1
@ -170,9 +173,9 @@ PUT /_plugins/_replication/<follower-index>/_update
Introduced 1.1
{: .label .label-purple }
Automatically starts replication on indices matching a specified pattern. Newly created indices on the remote cluster that match one of the specified patterns will be automatically configured as follower indices. You can also use this API to update existing auto-follow patterns.
Automatically starts replication on indices matching a specified pattern. If a new index on the leader cluster matches the pattern, OpenSearch automatically creates a follower index and begins replication. You can also use this API to update existing replication rules.
Run this command on the follower cluster.
Send this request to the follower cluster.
Make sure to note the names of all auto-follow patterns after you create them. The replication plugin currently does not include an API operation to retrieve a list of existing patterns.
{: .tip }
@ -215,7 +218,7 @@ Introduced 1.1
Deletes the specified replication rule. This operation prevents any new indices from being replicated but does not stop existing replication that the rule has already initiated.
Run this command on the follower cluster.
Send this request to the follower cluster.
#### Request

4
_replication-plugin/auto-follow.md
View File

@ -42,7 +42,7 @@ curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://loc
}'
```
If the security plugin is disabled, you can leave out the `use_roles` parameter. If it's enabled, however, you need to specify the leader and follower cluster roles that OpenSearch will use to authenticate the request. This example uses `all_access` for simplicity, but we recommend creating a replication user on each cluster and [mapping it accordingly]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles).
If the security plugin is disabled, you can leave out the `use_roles` parameter. If it's enabled, however, you need to specify the leader and follower cluster roles that OpenSearch uses to authenticate requests. This example uses `all_access` for simplicity, but we recommend creating a replication user on each cluster and [mapping it accordingly]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles).
{: .tip }
To test the rule, create a matching index on the leader cluster:
@ -64,7 +64,7 @@ yellow open movies-0001 kHOxYYHxRMeszLjTD9rvSQ 1 1 0
## Delete a replication rule
When you delete a replication rule, OpenSearch stops replicating *new* indices that match the pattern, but replication of existing indices that the rule previously created will continue. If you need to stop existing replication activity, use the [stop replication API operation]({{site.url}}{{site.baseurl}}/replication-plugin/api/#stop-replication).
When you delete a replication rule, OpenSearch stops replicating *new* indices that match the pattern, but existing indices that the rule previously created continue to replicate. If you need to stop existing replication activity, use the [stop replication API operation]({{site.url}}{{site.baseurl}}/replication-plugin/api/#stop-replication).
```bash
curl -XDELETE -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/_autofollow?pretty' -d '

40
_replication-plugin/get-started.md
View File

@ -6,13 +6,13 @@ nav_order: 10
# Get started with cross-cluster replication
With cross-cluster replication, you index data to a leader index and that data is replicated to one or more read-only follower indices. All subsequent operations on the leader are replicated on the follower, such as creating, updating, or deleting documents.
With cross-cluster replication, you index data to a leader index, and OpenSearch replicates that data to one or more read-only follower indices. All subsequent operations on the leader are replicated on the follower, such as creating, updating, or deleting documents.
## Prerequisites
Cross-cluster replication has the following prerequisites:
- Install the replication plugin on all nodes of both the leader and the follower cluster.
- If you've overridden `node.roles` in opensearch.yml on the remote cluster, make sure it also includes the `remote_cluster_client` role:
- Both the leader and follower cluster must have the replication plugin installed.
- If you've overridden `node.roles` in `opensearch.yml` on the remote cluster, make sure it also includes the `remote_cluster_client` role:
```yaml
node.roles: [<other_roles>, remote_cluster_client]
@ -20,11 +20,11 @@ Cross-cluster replication has the following prerequisites:
## Permissions
Make sure the security plugin is either enabled on both clusters or disabled on both clusters. If you disabled the security plugin, you can skip this section.
Make sure the security plugin is either enabled on both clusters or disabled on both clusters. If you disabled the security plugin, you can skip this section. However, we strongly recommend enabling the security plugin in production scenarios.
If the security plugin is enabled, non-admin users need to be mapped to the appropriate permissions in order to perform replication actions. For index and cluster-level permissions requirements, see [Cross-cluster replication permissions]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/).
In addition, add the following setting to opensearch.yml on the leader cluster so it allows connections from the follower cluster:
In addition, add the following setting to `opensearch.yml` on the leader cluster so it allows connections from the follower cluster:
```yml
plugins.security.nodes_dn_dynamic_config_enabled: true
@ -32,6 +32,9 @@ plugins.security.nodes_dn_dynamic_config_enabled: true
## Example setup
The following example demonstrates how to replicate data between two single-node clusters: `leader-cluster` on port 9201, and `follower-cluster` on port 9200.
{% comment %}
Save this sample file as `docker-compose.yml` and run `docker-compose up` to start two single-node clusters on the same network:
```yml
@ -86,6 +89,8 @@ networks:
After the clusters start, verify the names of each:
{% endcomment %}
```bash
curl -XGET -u 'admin:admin' -k 'https://localhost:9201'
{
@ -102,6 +107,8 @@ curl -XGET -u 'admin:admin' -k 'https://localhost:9200'
}
```
{% comment %}
For this example, use port 9201 (`replication-node1`) as the leader and port 9200 (`replication-node2`) as the follower cluster.
To get the IP address for the leader cluster, first identify its container ID:
@ -119,10 +126,13 @@ Then get that container's IP address:
docker inspect --format='{% raw %}{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}{% endraw %}' 731f5e8b0f4b
172.22.0.3
```
{% endcomment %}
## Set up a cross-cluster connection
On the follower cluster, add the leader cluster name and the IP address (with port 9300) for each seed node. In this case, you only have one seed node:
Cross-cluster replication follows a "pull" model, so most changes occur on the follower cluster, not the leader cluster.
On the follower cluster, add the leader cluster name and the IP address (with port 9300) for each seed node. Because this is a single-node cluster, you only have one seed node:
```bash
curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_cluster/settings?pretty' -d '
@ -141,15 +151,13 @@ curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://loca
## Start replication
To get started, create an index called `leader-01` on the remote (leader) cluster:
To get started, create an index called `leader-01` on the leader cluster:
```bash
curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9201/leader-01?pretty'
```
Start replication of that index from the follower cluster. Starting replication creates a follower index from scratch; you can't convert an existing index to a follower index.
Provide the leader cluster and index that you want to replicate:
Then start replication of that index from the follower cluster. In the request body, provide the leader cluster and index, along with the security roles that you want to use:
```bash
curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_start?pretty' -d '
@ -163,10 +171,12 @@ curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://loca
}'
```
If the security plugin is disabled, you can leave out the `use_roles` parameter. If it's enabled, however, you need to specify the leader and follower cluster roles that OpenSearch will use to authenticate the request. This example uses `all_access` for simplicity, but we recommend creating a replication user on each cluster and [mapping it accordingly]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles).
If the security plugin is disabled, omit the `use_roles` parameter. If it's enabled, however, you must specify the leader and follower cluster roles that OpenSearch will use to authenticate the request. This example uses `all_access` for simplicity, but we recommend creating a replication user on each cluster and [mapping it accordingly]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles).
{: .tip }
This command creates an identical read-only index named "follower-01" on the local cluster that continuously stays updated with changes to the "leader-01" index on the remote cluster.
This command creates an identical read-only index named `follower-01` on the local cluster that continuously stays updated with changes to the `leader-01` index on the remote cluster. Starting replication creates a follower index from scratch; you can't convert an existing index to a follower index.
## Confirm replication
After replication starts, get the status:
@ -187,7 +197,7 @@ curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/fol
}
```
## Confirm replication
Possible statuses are `SYNCING`, `BOOTSTRAPING`, `PAUSED`, and `REPLICATION NOT IN PROGRESS`. The leader and follower checkpoint values increment with each change and illustrate how many updates the follower is behind the leader. If the indices are fully synced, the values are the same.
To confirm that replication is actually happening, add a document to the leader index:
@ -244,8 +254,6 @@ curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://loc
When replication resumes, the follower index picks up any changes that were made to the leader index while replication was paused.
If you don't resume replication within 12 hours, replication stops completely and the follower index is converted to a standard index.
## Stop replication
Terminate replication of a specified index from the follower cluster:
@ -254,7 +262,7 @@ Terminate replication of a specified index from the follower cluster:
curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_stop' -d '{}'
```
When you stop replication, the follower index un-follows the leader and becomes a standard index that you can write to. You can't restart replication after it's been terminated.
When you stop replication, the follower index un-follows the leader and becomes a standard index that you can write to. You can't restart replication after stopping it.
Get the status to confirm that the index is no longer being replicated:

13
_replication-plugin/index.md
View File

@ -8,12 +8,15 @@ has_children: false
# Cross-cluster replication
The cross-cluster replication plugin lets you replicate indices, mappings, and metadata from one OpenSearch cluster to another. It follows an active-passive replication model where the follower index (where the data is replicated) pulls data from the leader (source) index.
The cross-cluster replication plugin lets you replicate indices, mappings, and metadata from one OpenSearch cluster to another. Cross-cluster replication has the following benefits:
- By replicating your indices, you ensure that you can continue to handle search requests in the event of an outage.
- Replicating data across geographically distant data centers minimizes the distance between the data and the application server, reducing expensive latencies.
- You can replicate data from multiple smaller clusters to a centralized reporting cluster, which is useful when it's inefficient to query across a large network.
The replication plugin supports replication of indices using wildcard pattern matching and provides commands to pause, resume, and stop replication. Once replication starts on an index, it initiates a persistent background task on the primary shard on the follower cluster that continuously polls corresponding shards from the leader cluster for updates.
Replication follows an active-passive model where the follower index (where the data is replicated) pulls data from the leader (remote) index.
The replication plugin integrates with the security plugin so you can encrypt cross-cluster traffic with node-to-node encryption and control access to replication activities.
To start, see [Get started with cross-cluster replication]({{site.url}}{{site.baseurl}}/replication-plugin/get-started/).
The replication plugin supports replication of indices using wildcard pattern matching and provides commands to pause, resume, and stop replication. Once replication starts on an index, it initiates persistent background tasks on all primary shards on the follower cluster, which continuously poll corresponding shards from the leader cluster for updates.
You can use the replication plugin with the security plugin to encrypt cross-cluster traffic with node-to-node encryption and control access to replication activities.
To start, see [Get started with cross-cluster replication]({{site.url}}{{site.baseurl}}/replication-plugin/get-started/).

22
_replication-plugin/permissions.md
View File

@ -14,24 +14,26 @@ Because cross-cluster replication involves multiple clusters, it's possible that
- Security plugin enabled only for TLS on both clusters (`plugins.security.ssl_only`)
- Security plugin absent or disabled on both clusters (not recommended)
You can enable node-to-node encryption on both the leader and the follower cluster to ensure that replication traffic between the clusters is encrypted.
Enable node-to-node encryption on both the leader and the follower cluster to ensure that replication traffic between the clusters is encrypted.
## Basic permissions
In order for non-admin users to perform replication activities, they need to be mapped to the appropriate permissions.
In order for non-admin users to perform replication activities, they be mapped to the appropriate permissions.
The security plugin has two built-in roles that cover most replication use cases: `cross_cluster_replication_leader_full_access` which provides replication permissions on the leader cluster, and `cross_cluster_replication_follower_full_access` which provides replication permissions on the follower cluster. For descriptions of each, see [Predefined roles]({{site.url}}{{site.baseurl}}/security-plugin/access-control/users-roles#predefined-roles).
The security plugin has two built-in roles that cover most replication use cases: `cross_cluster_replication_leader_full_access`, which provides replication permissions on the leader cluster, and `cross_cluster_replication_follower_full_access`, which provides replication permissions on the follower cluster. For descriptions of each, see [Predefined roles]({{site.url}}{{site.baseurl}}/security-plugin/access-control/users-roles#predefined-roles).
If you don't want to use the default roles, you can combine individual replication [permissions]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#replication-permissions) to meet your needs. Most permissions correspond to specific REST API operations. For example, the `indices:admin/plugins/replication/index/pause` permission lets you pause replication.
## Map the leader and follower cluster roles
The [start replication]({{site.url}}{{site.baseurl}}/replication-plugin/api/#start-replication) and [create replication rule]({{site.url}}{{site.baseurl}}/replication-plugin/api/#start-replication) are special cases because they involve background processes that require permissions on both the leader and the follower cluster. Therefore, when you perform one of these actions, you need to explicitly pass the `leader_cluster_role` and
`follower_cluster_role` in the request, which OpenSearch will then use in all backend replication tasks.
associates roles passed in the request to these replication jobs to run in the background
To enable non-admins to start replication and create replication rules, create an identical user on each cluster (for example, `replication_user`) and map them to the `cross_cluster_replication_leader_full_access` role on the remote cluster and `cross_cluster_replication_follower_full_access` on the follower cluster. For instructions, see [Map users to roles]({{site.url}}{{site.baseurl}}/access-control/users-roles/#map-users-to-roles).
The [start replication]({{site.url}}{{site.baseurl}}/replication-plugin/api/#start-replication) and [create replication rule]({{site.url}}{{site.baseurl}}/replication-plugin/api/#create-replication-rule) operations are special cases. They involve background processes on the leader and follower clusters that must be associated with roles. When you perform one of these actions, you must explicitly pass the `leader_cluster_role` and
`follower_cluster_role` in the request, which OpenSearch then uses in all backend replication tasks.
You can then pass those roles into the request along with the appropriate credentials:
To enable non-admins to start replication and create replication rules, create an identical user on each cluster (for example, `replication_user`) and map them to the `cross_cluster_replication_leader_full_access` role on the remote cluster and `cross_cluster_replication_follower_full_access` on the follower cluster. For instructions, see [Map users to roles]({{site.url}}{{site.baseurl}}/security-plugin/access-control/users-roles/#map-users-to-roles).
Then add those roles to the request, and sign it with the appropriate credentials:
```bash
curl -XPUT -k -H 'Content-Type: application/json' -u 'replication_user:password' 'https://localhost:9200/_plugins/_replication/follower-01/_start?pretty' -d '
@ -45,7 +47,7 @@ curl -XPUT -k -H 'Content-Type: application/json' -u 'replication_user:password'
}'
```
You can instead choose to create your own leader and follower cluster roles to meet your needs, but we recommend using the default roles.
You can create your own, custom leader and follower cluster roles using individual permissions, but we recommend using the default roles, which are a good fit for most use cases.
## Replication permissions
@ -53,7 +55,7 @@ The following sections list the available index and cluster-level permissions fo
### Follower cluster
Users can have the following permissions for the follower cluster:
The security plugin supports these permissions for the follower cluster:
```
indices:admin/plugins/replication/index/setup/validate
@ -69,7 +71,7 @@ cluster:admin/plugins/replication/autofollow/update
### Leader cluster
Users can have the following permissions for the leader cluster:
The security plugin supports these permissions for the leader cluster:
```
indices:admin/plugins/replication/validate

14
_replication-plugin/settings.md
View File

@ -10,13 +10,13 @@ The replication plugin adds several settings to the standard OpenSearch cluster
The settings are dynamic, so you can change the default behavior of the plugin without restarting your cluster.
You can mark settings as `persistent` or `transient`.
For example, to update the retention period of the result index:
For example, to update how often the follower cluster polls the leader cluster for updates:
```json
PUT _cluster/settings
{
"persistent": {
"plugins.replication.indices.recovery.parallel_chunks": "8"
"plugins.replication.follower.metadata_sync_interval": "30s"
}
}
```
@ -25,8 +25,10 @@ These settings manage the resources consumed by remote recoveries. We don’t re
Setting | Default | Description
:--- | :--- | :---
`plugins.replication.indices.recovery.chunk_size` | 1MB | The chunk size requested by the follower cluster during file transfer. Specify the chunk size as a value and unit, for example, 10MB, 5KB.
`plugins.replication.indices.recovery.parallel_chunks` | 5 | The number of file chunk requests that can be sent in parallel for each recovery.
`plugins.replication.indices.recovery.request_timeout` | 60s | The amount of time to wait for individual network requests during the remote recovery process. A single action timeout can cause recovery to fail.
`plugins.replication.indices.recovery.activity_timeout` | 5m | The amount of time to wait for recovery activity. If the leader cluster doesn't receive recovery requests from the follower for this amount of time, it closes the in-memory resources needed to supply data to the follower during recovery.
`plugins.replication.follower.index.recovery.chunk_size` | 10MB | The chunk size requested by the follower cluster during file transfer. Specify the chunk size as a value and unit, for example, 10MB, 5KB. See [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/).
`plugins.replication.follower.index.recovery.max_concurrent_file_chunks` | 4 | The number of file chunk requests that can be sent in parallel for each recovery.
`plugins.replication.follower.index.ops_batch_size` | 5000 | The number of operations that can be fetched at a time during the syncing phase of replication.
`plugins.replication.follower.concurrent_readers_per_shard` | 2 | The number of concurrent requests from the follower cluster per shard during the syncing phase of replication.
`plugins.replication.autofollow.fetch_poll_interval` | 30s | How often auto-follow tasks poll the leader cluster for new matching indices.
`plugins.replication.follower.metadata_sync_interval` | 60s | How often the follower cluster polls the leader cluster for updated index metadata.