Adds documentation page for remote-backed storage (#1179)

* Created page based on structure of PR 1163

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Added prerequisite steps for enabling the feature and its dependency

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Adding more

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Adding more content

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Finishing up and ready for tech review

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Moved page per dev request

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Fixed header and changed heading sizes

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Ready for tech review

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

* Added link to segment replication

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>

Signed-off-by: JeffH-AWS <jeffhuss@amazon.com>
This commit is contained in:
Jeff Huss 2022-09-14 09:11:14 -07:00 committed by GitHub
parent 9eb3a237b9
commit f805a28bf7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 120 additions and 0 deletions

120
_opensearch/remote.md Normal file
View File

@ -0,0 +1,120 @@
---
layout: default
title: Remote-backed storage
nav_order: 19
---
# Remote-backed storage
Remote-backed storage offers OpenSearch users a new way to protect against data loss by automatically creating backups of all index transactions and sending them to a supported cloud storage service. In order to expose this feature, segment replication must also be enabled. See [Segment replication]({{site.url}}{{site.baseurl}}/opensearch/segment-replication/) for additional information.
Remote-backed storage is an experimental feature. Therefore, we do not recommend the use of remote-backed storage in a production environment. For updates on the progress of remote-backed storage, or if you want leave feedback that could help improve the feature, refer to the issue on [GitHub](https://github.com/opensearch-project/OpenSearch/issues/1968).
{: .note}
## Enable the feature
In order to create new indexes with remote-backed storage enabled, you must first enable these features by adding the correct properties to `run.gradle` before building OpenSearch. See the [developer guide](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md) for information about how Gradle is used to build OpenSearch.
Add the following properties to `run.gradle` to enable the feature:
```bash
testClusters {
runTask {
testDistribution = 'archive'
if (numZones > 1) numberOfZones = numZones
if (numNodes > 1) numberOfNodes = numNodes
systemProperty 'opensearch.experimental.feature.replication_type.enabled', 'true'
systemProperty 'opensearch.experimental.feature.remote_store.enabled', 'true'
}
}
```
Segment replication must also be enabled in order to use remote-backed storage.
{: .note}
After building OpenSearch with these properties, you must enable the feature for all nodes in your cluster. This can be done by modifying a `config/jvm.options`, or by defining `OPENSEARCH_JAVA_OPS` from the command line.
### Option 1: Modify jvm.options
Add the following lines to `config/jvm.options` before starting the OpenSearch process to enable the feature and its dependency:
```
-Dopensearch.experimental.feature.replication_type.enabled=true
-Dopensearch.experimental.feature.remote_store.enabled=true
```
Run OpenSearch
```bash
./bin/opensearch
```
### Option 2: Enable from an environment variable
As an alternative to directly modifying `config/jvm.options`, you can define the properties by using an environment variable. This can be done in a single command when you start OpenSearch or by defining the variable with `export`.
To add these flags in-line when starting OpenSearch:
```bash
OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.replication_type.enabled=true -Dopensearch.experimental.feature.remote_store.enabled=true" ./opensearch-{{site.opensearch_version}}/bin/opensearch
```
If you want to define the environment variable separately, prior to running OpenSearch:
```bash
export OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.replication_type.enabled=true -Dopensearch.experimental.feature.remote_store.enabled=true"
./bin/opensearch
```
## Register a remote repository
Now that your deployment is running with the feature flags enabled, the next step is to register a remote repository where backups will be stored. See [Register repository]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#register-repository) for more information.
## Create an index
Remote-backed storage is enabled for an index when it is created. This feature cannot be enabled for indexes that already exist.
When creating the index, include the `remote_store` property to enable the feature and specify a target repository:
```bash
curl -X PUT "https://localhost:9200/my-index?pretty" -ku admin:admin -H 'Content-Type: application/json' -d'
{
"settings": {
"index": {
"number_of_shards": 1,
"number_of_replicas": 0,
"replication": {
"type": "SEGMENT"
},
"remote_store": {
"enabled": true,
"repository": "my-repo-1"
}
}
}
}
'
```
All data that is added to the index will also be uploaded to the remote storage once it is committed.
### Restoring from a backup
To restore an index from a remote backup, such as in the event of a node failure, you must first close the index:
```bash
curl -X POST "https://localhost:9200/my-index/_close" -ku admin:admin
```
Restore the index from the backup stored on the remote repository:
```bash
curl -X POST "https://localhost:9200/_remotestore/_restore" -ku admin:admin -H 'Content-Type: application/json' -d'
{
"indices": ["my-index"]
}
'
```
If the security plugin is enabled, a user must have the `cluster:admin/remotestore/restore` permission. See [Access control](/security-plugin/access-control/index/) for information about configuring user permissions.
{: .note}