Publish segment replication experimental feature documentation (#1163)
* segment replication documentation Signed-off-by: ariamarble <armarble@amazon.com> * changes from tech review Signed-off-by: ariamarble <armarble@amazon.com> * renamed file type, fixed various wording and formatting Signed-off-by: ariamarble <armarble@amazon.com> * made suggested edits Signed-off-by: ariamarble <armarble@amazon.com> * edit corrections Signed-off-by: ariamarble <armarble@amazon.com> Signed-off-by: ariamarble <armarble@amazon.com>
This commit is contained in:
Aria Marble
GitHub
parent
1f54720013
commit
e2b82dd75e
|
|
@ -0,0 +1,84 @@
|
|||
---
|
||||
layout: default
|
||||
title: Segment Replication Configuration
|
||||
nav_order: 12
|
||||
parent: Segment Replication Feature
|
||||
grand_parent: Opensearch
|
||||
---
|
||||
|
||||
## Segment replication configuration
|
||||
|
||||
To enable the segment replication type, reference the steps below.
|
||||
|
||||
### Enabling the feature flag
|
||||
|
||||
There are several methods for enabling segment replication, depending on the install type. You will also need to set the replication strategy to `SEGMENT` when creating the index.
|
||||
|
||||
#### Enable on a node using a tarball install
|
||||
|
||||
The flag is toggled using a new jvm parameter that is set either in `OPENSEARCH_JAVA_OPTS` or in config/jvm.options.
|
||||
|
||||
1. Option 1: Update config/jvm.options by adding the following line:
|
||||
|
||||
````json
|
||||
-Dopensearch.experimental.feature.replication_type.enabled=true
|
||||
````
|
||||
|
||||
1. Option 2: Use the `OPENSEARCH_JAVA_OPTS` environment variable:
|
||||
|
||||
````json
|
||||
export OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.replication_type.enabled=true"
|
||||
````
|
||||
1. Option 3: For developers using Gradle, update run.gradle by adding the following lines:
|
||||
|
||||
````json
|
||||
testClusters {
|
||||
runTask {
|
||||
testDistribution = 'archive'
|
||||
if (numZones > 1) numberOfZones = numZones
|
||||
if (numNodes > 1) numberOfNodes = numNodes
|
||||
systemProperty 'opensearch.experimental.feature.replication_type.enabled', 'true'
|
||||
}
|
||||
}
|
||||
````
|
||||
|
||||
#### Enable with Docker containers
|
||||
|
||||
If you're running Docker, add the following line to docker-compose.yml underneath the `opensearch-node` and `environment` section:
|
||||
|
||||
````json
|
||||
OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.replication_type.enabled=true" # Enables segment replication
|
||||
````
|
||||
|
||||
#### Setting the replication strategy on the index
|
||||
|
||||
To set the replication strategy to segment replication, create an index with replication.type set to `SEGMENT`:
|
||||
|
||||
````json
|
||||
PUT /my-index1
|
||||
{
|
||||
"settings": {
|
||||
"index": {
|
||||
"replication.type": "SEGMENT"
|
||||
}
|
||||
}
|
||||
}
|
||||
````
|
||||
|
||||
### Known limitations
|
||||
|
||||
Segment replication is an experimental feature. Therefore, we do not recommend the use of segment replication in a production environment. For updates on the progress of segment replication or if you want to leave feedback that could help improve the feature, see the [Segment Replication Git Issue](https://github.com/opensearch-project/OpenSearch/issues/2229).
|
||||
{: .note }
|
||||
|
||||
1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685).
|
||||
1. Rolling upgrades are currently not supported. Full cluster restarts are required when upgrading indexes using segment replication. [Issue 3881](https://github.com/opensearch-project/OpenSearch/issues/3881).
|
||||
1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters.
|
||||
1. Increased network congestion on primary shards. [Issue - Optimize network bandwidth on primary shards](https://github.com/opensearch-project/OpenSearch/issues/4245).
|
||||
1. Shard allocation algorithms have not been updated to evenly spread primary shards across nodes.
|
||||
1. Integration with remote-backed storage as the source of replication is [currently unsupported](https://github.com/opensearch-project/OpenSearch/issues/4448).
|
||||
|
||||
### Further resources regarding segment replication
|
||||
|
||||
1. [Known issues](https://github.com/opensearch-project/OpenSearch/issues/2194).
|
||||
1. Steps for testing (link coming soon).
|
||||
1. Segment replication blog post (link coming soon).
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
---
|
||||
layout: default
|
||||
title: Segment Replication Feature
|
||||
nav_order: 122
|
||||
parent: Opensearch
|
||||
has_children: true
|
||||
redirect_from:
|
||||
- /opensearch/segment-replication/
|
||||
---
|
||||
|
||||
## Segment replication
|
||||
|
||||
OpenSearch 2.3 contains an experimental version of segment replication.
|
||||
|
||||
With segment replication, segment files are copied across shards instead of documents being indexed on each shard copy. This improves indexing throughput and lowers resource utilization at the expense of increased network utilization.
|
||||
|
||||
As an experimental feature, segment replication will be behind a feature flag and must be enabled on **each node** of a cluster and pass a new setting during index creation.
|
||||
{: .note }
|
||||
|
||||
### Potential use cases
|
||||
|
||||
- Users who have very high write loads but do not have high search requirements and are comfortable with longer refresh times.
|
||||
- Users with very high loads who want to add new nodes, as you do not need to index all nodes when adding a new node to the cluster.
|
||||
|
||||
This is the first step in a series of features designed to decouple reads and writes in order to lower compute costs.
|
||||
Loading…
Reference in New Issue