From 9414701fda435d76c3cd9a3fd0395ab6f8475953 Mon Sep 17 00:00:00 2001 From: Aria Marble <111301581+ariamarble@users.noreply.github.com> Date: Wed, 1 Mar 2023 14:52:22 -0800 Subject: [PATCH] Update the create snapshot repository documentation (#3116) * Update create snapshot repository documentation Signed-off-by: ariamarble * Apply suggestions from doc review review Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * made additional review changes Signed-off-by: ariamarble * Apply suggestions from editorial review Co-authored-by: Nathan Bower --------- Signed-off-by: ariamarble Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Nathan Bower --- _api-reference/snapshots/create-repository.md | 64 +++++++++------ .../snapshots/snapshot-restore.md | 78 +++++++------------ 2 files changed, 71 insertions(+), 71 deletions(-) diff --git a/_api-reference/snapshots/create-repository.md b/_api-reference/snapshots/create-repository.md index b0313e1b..a0672f8c 100644 --- a/_api-reference/snapshots/create-repository.md +++ b/_api-reference/snapshots/create-repository.md @@ -5,42 +5,65 @@ parent: Snapshot APIs nav_order: 1 --- -## Register or update snapshot repository +# Registering or updating a snapshot repository -Registers a new repository to store snapshots, or updates information for an existing repository. Repositories can be of two types: +You can register a new repository in which to store snapshots or update information for an existing repository by using the snapshots API. -* File system (`fs`) +There are two types of snapshot repositories: -* Amazon S3 bucket (`s3`) +* File system (`fs`): For instructions on creating an `fs` repository, see [Register repository shared file system]({{site.url}}{{site.baseurl}}/availability-and-recovery/snapshots/snapshot-restore/#shared-file-system). +* Amazon Simple Storage Service (Amazon S3) bucket (`s3`): For instructions on creating an `s3` repository, see [Register repository Amazon S3]({{site.url}}{{site.baseurl}}/availability-and-recovery/snapshots/snapshot-restore/#amazon-s3). -To learn more about repositories, see [Register repository]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#register-repository). +For instructions on creating a repository, see [Register repository]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#register-repository). -### Path parameters +## Path and HTTP methods + +``` +POST /_snapshot/my-first-repo/ +PUT /_snapshot/my-first-repo/ +``` + +## Path parameters Parameter | Data type | Description :--- | :--- | :--- -repository | String | Repostory name. | +repository | String | Repository name | -### Request fields +## Request parameters -Allowable parameters depend on the repository type. +Request parameters depend on the type of repository: `fs` or `s3`. -#### fs repository type +### fs repository -Setting | Description +Request field | Description :--- | :--- -`location` | The shared file system for snapshots. Required. +`location` | The file system directory for snapshots, such as a mounted directory from a file server or a Samba share. Must be accessible by all nodes. Required. `chunk_size` | Breaks large files into chunks during snapshot operations (e.g. `64mb`, `1gb`), which is important for cloud storage providers and far less important for shared file systems. Default is `null` (unlimited). Optional. `compress` | Whether to compress metadata files. This setting does not affect data files, which might already be compressed, depending on your index settings. Default is `false`. Optional. `max_restore_bytes_per_sec` | The maximum rate at which snapshots restore. Default is 40 MB per second (`40m`). Optional. `max_snapshot_bytes_per_sec` | The maximum rate at which snapshots take. Default is 40 MB per second (`40m`). Optional. `readonly` | Whether the repository is read-only. Useful when migrating from one cluster (`"readonly": false` when registering) to another cluster (`"readonly": true` when registering). Optional. -#### s3 repository type +#### Example request -Setting | Description +The following example registers an `fs` repository using the local directory `/mnt/snapshots` as `location`. + +```json +PUT /_snapshot/my-fs-repository +{ + "type": "fs", + "settings": { + "location": "/mnt/snapshots" + } +} +``` +{% include copy-curl.html %} + +### s3 repository + +Request field | Description :--- | :--- -`base_path` | The path within the bucket where you want to store snapshots (e.g. `my/snapshot/directory`). Optional. If not specified, snapshots are stored in the bucket root. +`base_path` | The path within the bucket in which you want to store snapshots (for example, `my/snapshot/directory`). Optional. If not specified, snapshots are stored in the S3 bucket root. `bucket` | Name of the S3 bucket. Required. `buffer_size` | The threshold beyond which chunks (of `chunk_size`) should be broken into pieces (of `buffer_size`) and sent to S3 using a different API. Default is the smaller of two values: 100 MB or 5% of the Java heap. Valid values are between `5mb` and `5gb`. We don't recommend changing this option. `canned_acl` | S3 has several [canned ACLs](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl) that the `repository-s3` plugin can add to objects as it creates them in S3. Default is `private`. Optional. @@ -53,18 +76,15 @@ Setting | Description `server_side_encryption` | Whether to encrypt snapshot files in the S3 bucket. This setting uses AES-256 with S3-managed keys. See [Protecting data using server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html). Default is false. Optional. `storage_class` | Specifies the [S3 storage class](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html) for the snapshots files. Default is `standard`. Do not use the `glacier` and `deep_archive` storage classes. Optional. +For the `base_path` parameter, do not enter the `s3://` prefix when entering your S3 bucket details. Only the name of the bucket is required. +{: .note} + #### Example request -The following requests register or update a repository called `my-first-repo`. Both require a request body as described in [Request fields](#request-fields). - -* `POST _snapshot/my-first-repo/` registers a snapshot repository. - -* `PUT _snapshot/my-first-repo/` registers a snapshot repository if it does not exist; otherwise, updates information about the repository. - The following request registers a new S3 repository called `my-opensearch-repo` in an existing bucket called `my-open-search-bucket`. By default, all snapshots are stored in the `my/snapshot/directory`. ```json -PUT _snapshot/my-opensearch-repo +PUT /_snapshot/my-opensearch-repo { "type": "s3", "settings": { diff --git a/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md b/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md index 6ac3c58c..c51a5810 100644 --- a/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md +++ b/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md @@ -53,7 +53,7 @@ Before you can take a snapshot, you have to "register" a snapshot repository. A 1. Then register the repository using the REST API: ```json - PUT _snapshot/my-fs-repository + PUT /_snapshot/my-fs-repository { "type": "fs", "settings": { @@ -61,26 +61,9 @@ Before you can take a snapshot, you have to "register" a snapshot repository. A } } ``` + {% include copy-curl.html %} - If the request is successful, the response from OpenSearch is minimal: - - ```json - { - "acknowledged": true - } - ``` - -You probably only need to specify `location`, but the following table summarizes the options: - -Request fields | Description -:--- | :--- -`location` | The shared file system for snapshots. Required. -`chunk_size` | Breaks large files into chunks during snapshot operations (e.g. `64mb`, `1gb`), which is important for cloud storage providers and far less important for shared file systems. Default is `null` (unlimited). Optional. -`compress` | Whether to compress metadata files. This setting does not affect data files, which might already be compressed, depending on your index settings. Default is `false`. Optional. -`max_restore_bytes_per_sec` | The maximum rate at which snapshots restore. Default is 40 MB per second (`40m`). Optional. -`max_snapshot_bytes_per_sec` | The maximum rate at which snapshots take. Default is 40 MB per second (`40m`). Optional. -`readonly` | Whether the repository is read-only. Useful when migrating from one cluster (`"readonly": false` when registering) to another cluster (`"readonly": true` when registering). Optional. - +You will most likely not need to specify any parameters except for `location`. For allowed request parameters, see [Register or update snapshot repository API](https://opensearch.org/docs/latest/api-reference/snapshots/create-repository/). ### Amazon S3 @@ -179,8 +162,9 @@ Request fields | Description 1. If you changed `opensearch.yml`, you must restart each node in the cluster. Otherwise, you only need to reload secure cluster settings: ``` - POST _nodes/reload_secure_settings + POST /_nodes/reload_secure_settings ``` + {% include copy-curl.html %} 1. Create an S3 bucket if you don't already have one. To take snapshots, you need permissions to access the bucket. The following IAM policy is an example of those permissions: @@ -203,7 +187,7 @@ Request fields | Description 1. Register the repository using the REST API: ```json - PUT _snapshot/my-s3-repository + PUT /_snapshot/my-s3-repository { "type": "s3", "settings": { @@ -212,24 +196,9 @@ Request fields | Description } } ``` + {% include copy-curl.html %} -You probably don't need to specify anything but `bucket` and `base_path`, but the following table summarizes the options: - -Request fields | Description -:--- | :--- -`base_path` | The path within the bucket where you want to store snapshots (e.g. `my/snapshot/directory`). Optional. If not specified, snapshots are stored in the bucket root. -`bucket` | Name of the S3 bucket. Required. -`buffer_size` | The threshold beyond which chunks (of `chunk_size`) should be broken into pieces (of `buffer_size`) and sent to S3 using a different API. Default is the smaller of two values: 100 MB or 5% of the Java heap. Valid values are between `5mb` and `5gb`. We don't recommend changing this option. -`canned_acl` | S3 has several [canned ACLs](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl) that the `repository-s3` plugin can add to objects as it creates them in S3. Default is `private`. Optional. -`chunk_size` | Breaks files into chunks during snapshot operations (e.g. `64mb`, `1gb`), which is important for cloud storage providers and far less important for shared file systems. Default is `1gb`. Optional. -`client` | When specifying client settings (e.g. `s3.client.default.access_key`), you can use a string other than `default` (e.g. `s3.client.backup-role.access_key`). If you used an alternate name, change this value to match. Default and recommended value is `default`. Optional. -`compress` | Whether to compress metadata files. This setting does not affect data files, which might already be compressed, depending on your index settings. Default is `false`. Optional. -`max_restore_bytes_per_sec` | The maximum rate at which snapshots restore. Default is 40 MB per second (`40m`). Optional. -`max_snapshot_bytes_per_sec` | The maximum rate at which snapshots take. Default is 40 MB per second (`40m`). Optional. -`readonly` | Whether the repository is read-only. Useful when migrating from one cluster (`"readonly": false` when registering) to another cluster (`"readonly": true` when registering). Optional. -`server_side_encryption` | Whether to encrypt snapshot files in the S3 bucket. This setting uses AES-256 with S3-managed keys. See [Protecting data using server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html). Default is false. Optional. -`storage_class` | Specifies the [S3 storage class](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html) for the snapshots files. Default is `standard`. Do not use the `glacier` and `deep_archive` storage classes. Optional. - +You will most likely not need to specify any parameters except for `bucket` and `base_path`. For allowed request parameters, see [Register or update snapshot repository API](https://opensearch.org/docs/latest/api-reference/snapshots/create-repository/). ## Take snapshots @@ -241,13 +210,14 @@ You specify two pieces of information when you create a snapshot: The following snapshot includes all indices and the cluster state: ```json -PUT _snapshot/my-repository/1 +PUT /_snapshot/my-repository/1 ``` +{% include copy-curl.html %} You can also add a request body to include or exclude certain indices or specify other settings: ```json -PUT _snapshot/my-repository/2 +PUT /_snapshot/my-repository/2 { "indices": "opensearch-dashboards*,my-index*,-my-index-2016", "ignore_unavailable": true, @@ -255,6 +225,7 @@ PUT _snapshot/my-repository/2 "partial": false } ``` +{% include copy-curl.html %} Request fields | Description :--- | :--- @@ -266,7 +237,7 @@ Request fields | Description If you request the snapshot immediately after taking it, you might see something like this: ```json -GET _snapshot/my-repository/2 +GET /_snapshot/my-repository/2 { "snapshots": [{ "snapshot": "2", @@ -283,12 +254,14 @@ GET _snapshot/my-repository/2 }] } ``` +{% include copy-curl.html %} Note that the snapshot is still in progress. If you want to wait for the snapshot to finish before continuing, add the `wait_for_completion` parameter to your request. Snapshots can take a while to complete, so consider whether or not this option fits your use case: ``` PUT _snapshot/my-repository/3?wait_for_completion=true ``` +{% include copy-curl.html %} Snapshots have the following states: @@ -303,8 +276,9 @@ INCOMPATIBLE | The snapshot is incompatible with the version of OpenSearch runni You can't take a snapshot if one is currently in progress. To check the status: ``` -GET _snapshot/_status +GET /_snapshot/_status ``` +{% include copy-curl.html %} ## Restore snapshots @@ -312,25 +286,28 @@ GET _snapshot/_status The first step in restoring a snapshot is retrieving existing snapshots. To see all snapshot repositories: ``` -GET _snapshot/_all +GET /_snapshot/_all ``` +{% include copy-curl.html %} To see all snapshots in a repository: ``` -GET _snapshot/my-repository/_all +GET /_snapshot/my-repository/_all ``` +{% include copy-curl.html %} Then restore a snapshot: ``` -POST _snapshot/my-repository/2/_restore +POST /_snapshot/my-repository/2/_restore ``` +{% include copy-curl.html %} Just like when taking a snapshot, you can add a request body to include or exclude certain indices or specify some other settings: ```json -POST _snapshot/my-repository/2/_restore +POST /_snapshot/my-repository/2/_restore { "indices": "opensearch-dashboards*,my-index*", "ignore_unavailable": true, @@ -347,8 +324,9 @@ POST _snapshot/my-repository/2/_restore ] } ``` +{% include copy-curl.html %} -Request fields | Description +Request parameters | Description :--- | :--- `indices` | The indices you want to restore. You can use `,` to create a list of indices, `*` to specify an index pattern, and `-` to exclude certain indices. Don't put spaces between items. Default is all indices. `ignore_unavailable` | If an index from the `indices` list doesn't exist, whether to ignore it rather than fail the restore operation. Default is false. @@ -385,18 +363,20 @@ If you're using the security plugin, snapshots have some additional restrictions If a snapshot contains global state, you must exclude it when performing the restore. If your snapshot also contains the `.opendistro_security` index, either exclude it or list all the other indices you want to include: ```json -POST _snapshot/my-repository/3/_restore +POST /_snapshot/my-repository/3/_restore { "indices": "-.opendistro_security", "include_global_state": false } ``` +{% include copy-curl.html %} The `.opendistro_security` index contains sensitive data, so we recommend excluding it when you take a snapshot. If you do need to restore the index from a snapshot, you must include an admin certificate in the request: ```bash curl -k --cert ./kirk.pem --key ./kirk-key.pem -XPOST 'https://localhost:9200/_snapshot/my-repository/3/_restore?pretty' ``` +{% include copy-curl.html %} We strongly recommend against restoring `.opendistro_security` using an admin certificate because doing so can alter the security posture of the entire cluster. See [A word of caution]({{site.url}}{{site.baseurl}}/security-plugin/configuration/security-admin/#a-word-of-caution) for a recommended process to back up and restore your security plugin configuration. {: .warning} \ No newline at end of file