From f805a28bf7f737d4b5942e2f5fdd1ec840689954 Mon Sep 17 00:00:00 2001 From: Jeff Huss Date: Wed, 14 Sep 2022 09:11:14 -0700 Subject: [PATCH] Adds documentation page for remote-backed storage (#1179) * Created page based on structure of PR 1163 Signed-off-by: JeffH-AWS * Added prerequisite steps for enabling the feature and its dependency Signed-off-by: JeffH-AWS * Adding more Signed-off-by: JeffH-AWS * Adding more content Signed-off-by: JeffH-AWS * Finishing up and ready for tech review Signed-off-by: JeffH-AWS * Moved page per dev request Signed-off-by: JeffH-AWS * Fixed header and changed heading sizes Signed-off-by: JeffH-AWS * Ready for tech review Signed-off-by: JeffH-AWS * Added link to segment replication Signed-off-by: JeffH-AWS Signed-off-by: JeffH-AWS --- _opensearch/remote.md | 120 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 _opensearch/remote.md diff --git a/_opensearch/remote.md b/_opensearch/remote.md new file mode 100644 index 00000000..8d34383c --- /dev/null +++ b/_opensearch/remote.md @@ -0,0 +1,120 @@ +--- +layout: default +title: Remote-backed storage +nav_order: 19 +--- + +# Remote-backed storage + +Remote-backed storage offers OpenSearch users a new way to protect against data loss by automatically creating backups of all index transactions and sending them to a supported cloud storage service. In order to expose this feature, segment replication must also be enabled. See [Segment replication]({{site.url}}{{site.baseurl}}/opensearch/segment-replication/) for additional information. + +Remote-backed storage is an experimental feature. Therefore, we do not recommend the use of remote-backed storage in a production environment. For updates on the progress of remote-backed storage, or if you want leave feedback that could help improve the feature, refer to the issue on [GitHub](https://github.com/opensearch-project/OpenSearch/issues/1968). +{: .note} + +## Enable the feature + +In order to create new indexes with remote-backed storage enabled, you must first enable these features by adding the correct properties to `run.gradle` before building OpenSearch. See the [developer guide](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md) for information about how Gradle is used to build OpenSearch. + +Add the following properties to `run.gradle` to enable the feature: + +```bash +testClusters { + runTask { + testDistribution = 'archive' + if (numZones > 1) numberOfZones = numZones + if (numNodes > 1) numberOfNodes = numNodes + systemProperty 'opensearch.experimental.feature.replication_type.enabled', 'true' + systemProperty 'opensearch.experimental.feature.remote_store.enabled', 'true' + } +} +``` + +Segment replication must also be enabled in order to use remote-backed storage. +{: .note} + +After building OpenSearch with these properties, you must enable the feature for all nodes in your cluster. This can be done by modifying a `config/jvm.options`, or by defining `OPENSEARCH_JAVA_OPS` from the command line. + +### Option 1: Modify jvm.options + +Add the following lines to `config/jvm.options` before starting the OpenSearch process to enable the feature and its dependency: + +``` +-Dopensearch.experimental.feature.replication_type.enabled=true +-Dopensearch.experimental.feature.remote_store.enabled=true +``` + +Run OpenSearch + +```bash +./bin/opensearch +``` + +### Option 2: Enable from an environment variable + +As an alternative to directly modifying `config/jvm.options`, you can define the properties by using an environment variable. This can be done in a single command when you start OpenSearch or by defining the variable with `export`. + +To add these flags in-line when starting OpenSearch: + +```bash +OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.replication_type.enabled=true -Dopensearch.experimental.feature.remote_store.enabled=true" ./opensearch-{{site.opensearch_version}}/bin/opensearch +``` + +If you want to define the environment variable separately, prior to running OpenSearch: + +```bash +export OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.replication_type.enabled=true -Dopensearch.experimental.feature.remote_store.enabled=true" +./bin/opensearch +``` + +## Register a remote repository + +Now that your deployment is running with the feature flags enabled, the next step is to register a remote repository where backups will be stored. See [Register repository]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#register-repository) for more information. + +## Create an index + +Remote-backed storage is enabled for an index when it is created. This feature cannot be enabled for indexes that already exist. + +When creating the index, include the `remote_store` property to enable the feature and specify a target repository: + +```bash +curl -X PUT "https://localhost:9200/my-index?pretty" -ku admin:admin -H 'Content-Type: application/json' -d' +{ + "settings": { + "index": { + "number_of_shards": 1, + "number_of_replicas": 0, + "replication": { + "type": "SEGMENT" + }, + "remote_store": { + "enabled": true, + "repository": "my-repo-1" + } + } + } +} +' +``` + +All data that is added to the index will also be uploaded to the remote storage once it is committed. + +### Restoring from a backup + +To restore an index from a remote backup, such as in the event of a node failure, you must first close the index: + +```bash +curl -X POST "https://localhost:9200/my-index/_close" -ku admin:admin +``` + +Restore the index from the backup stored on the remote repository: + +```bash +curl -X POST "https://localhost:9200/_remotestore/_restore" -ku admin:admin -H 'Content-Type: application/json' -d' +{ + "indices": ["my-index"] +} +' +``` + +If the security plugin is enabled, a user must have the `cluster:admin/remotestore/restore` permission. See [Access control](/security-plugin/access-control/index/) for information about configuring user permissions. +{: .note} \ No newline at end of file