Adds documentation page for remote-backed storage (#1179)
* Created page based on structure of PR 1163 Signed-off-by: JeffH-AWS <jeffhuss@amazon.com> * Added prerequisite steps for enabling the feature and its dependency Signed-off-by: JeffH-AWS <jeffhuss@amazon.com> * Adding more Signed-off-by: JeffH-AWS <jeffhuss@amazon.com> * Adding more content Signed-off-by: JeffH-AWS <jeffhuss@amazon.com> * Finishing up and ready for tech review Signed-off-by: JeffH-AWS <jeffhuss@amazon.com> * Moved page per dev request Signed-off-by: JeffH-AWS <jeffhuss@amazon.com> * Fixed header and changed heading sizes Signed-off-by: JeffH-AWS <jeffhuss@amazon.com> * Ready for tech review Signed-off-by: JeffH-AWS <jeffhuss@amazon.com> * Added link to segment replication Signed-off-by: JeffH-AWS <jeffhuss@amazon.com> Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>
This commit is contained in:
parent
9eb3a237b9
commit
f805a28bf7
|
@ -0,0 +1,120 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: Remote-backed storage
|
||||||
|
nav_order: 19
|
||||||
|
---
|
||||||
|
|
||||||
|
# Remote-backed storage
|
||||||
|
|
||||||
|
Remote-backed storage offers OpenSearch users a new way to protect against data loss by automatically creating backups of all index transactions and sending them to a supported cloud storage service. In order to expose this feature, segment replication must also be enabled. See [Segment replication]({{site.url}}{{site.baseurl}}/opensearch/segment-replication/) for additional information.
|
||||||
|
|
||||||
|
Remote-backed storage is an experimental feature. Therefore, we do not recommend the use of remote-backed storage in a production environment. For updates on the progress of remote-backed storage, or if you want leave feedback that could help improve the feature, refer to the issue on [GitHub](https://github.com/opensearch-project/OpenSearch/issues/1968).
|
||||||
|
{: .note}
|
||||||
|
|
||||||
|
## Enable the feature
|
||||||
|
|
||||||
|
In order to create new indexes with remote-backed storage enabled, you must first enable these features by adding the correct properties to `run.gradle` before building OpenSearch. See the [developer guide](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md) for information about how Gradle is used to build OpenSearch.
|
||||||
|
|
||||||
|
Add the following properties to `run.gradle` to enable the feature:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
testClusters {
|
||||||
|
runTask {
|
||||||
|
testDistribution = 'archive'
|
||||||
|
if (numZones > 1) numberOfZones = numZones
|
||||||
|
if (numNodes > 1) numberOfNodes = numNodes
|
||||||
|
systemProperty 'opensearch.experimental.feature.replication_type.enabled', 'true'
|
||||||
|
systemProperty 'opensearch.experimental.feature.remote_store.enabled', 'true'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Segment replication must also be enabled in order to use remote-backed storage.
|
||||||
|
{: .note}
|
||||||
|
|
||||||
|
After building OpenSearch with these properties, you must enable the feature for all nodes in your cluster. This can be done by modifying a `config/jvm.options`, or by defining `OPENSEARCH_JAVA_OPS` from the command line.
|
||||||
|
|
||||||
|
### Option 1: Modify jvm.options
|
||||||
|
|
||||||
|
Add the following lines to `config/jvm.options` before starting the OpenSearch process to enable the feature and its dependency:
|
||||||
|
|
||||||
|
```
|
||||||
|
-Dopensearch.experimental.feature.replication_type.enabled=true
|
||||||
|
-Dopensearch.experimental.feature.remote_store.enabled=true
|
||||||
|
```
|
||||||
|
|
||||||
|
Run OpenSearch
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./bin/opensearch
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option 2: Enable from an environment variable
|
||||||
|
|
||||||
|
As an alternative to directly modifying `config/jvm.options`, you can define the properties by using an environment variable. This can be done in a single command when you start OpenSearch or by defining the variable with `export`.
|
||||||
|
|
||||||
|
To add these flags in-line when starting OpenSearch:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.replication_type.enabled=true -Dopensearch.experimental.feature.remote_store.enabled=true" ./opensearch-{{site.opensearch_version}}/bin/opensearch
|
||||||
|
```
|
||||||
|
|
||||||
|
If you want to define the environment variable separately, prior to running OpenSearch:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.replication_type.enabled=true -Dopensearch.experimental.feature.remote_store.enabled=true"
|
||||||
|
./bin/opensearch
|
||||||
|
```
|
||||||
|
|
||||||
|
## Register a remote repository
|
||||||
|
|
||||||
|
Now that your deployment is running with the feature flags enabled, the next step is to register a remote repository where backups will be stored. See [Register repository]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#register-repository) for more information.
|
||||||
|
|
||||||
|
## Create an index
|
||||||
|
|
||||||
|
Remote-backed storage is enabled for an index when it is created. This feature cannot be enabled for indexes that already exist.
|
||||||
|
|
||||||
|
When creating the index, include the `remote_store` property to enable the feature and specify a target repository:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X PUT "https://localhost:9200/my-index?pretty" -ku admin:admin -H 'Content-Type: application/json' -d'
|
||||||
|
{
|
||||||
|
"settings": {
|
||||||
|
"index": {
|
||||||
|
"number_of_shards": 1,
|
||||||
|
"number_of_replicas": 0,
|
||||||
|
"replication": {
|
||||||
|
"type": "SEGMENT"
|
||||||
|
},
|
||||||
|
"remote_store": {
|
||||||
|
"enabled": true,
|
||||||
|
"repository": "my-repo-1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
'
|
||||||
|
```
|
||||||
|
|
||||||
|
All data that is added to the index will also be uploaded to the remote storage once it is committed.
|
||||||
|
|
||||||
|
### Restoring from a backup
|
||||||
|
|
||||||
|
To restore an index from a remote backup, such as in the event of a node failure, you must first close the index:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST "https://localhost:9200/my-index/_close" -ku admin:admin
|
||||||
|
```
|
||||||
|
|
||||||
|
Restore the index from the backup stored on the remote repository:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST "https://localhost:9200/_remotestore/_restore" -ku admin:admin -H 'Content-Type: application/json' -d'
|
||||||
|
{
|
||||||
|
"indices": ["my-index"]
|
||||||
|
}
|
||||||
|
'
|
||||||
|
```
|
||||||
|
|
||||||
|
If the security plugin is enabled, a user must have the `cluster:admin/remotestore/restore` permission. See [Access control](/security-plugin/access-control/index/) for information about configuring user permissions.
|
||||||
|
{: .note}
|
Loading…
Reference in New Issue