From ebe1faeefd5718c7eb49700f1eaddad89628bbc8 Mon Sep 17 00:00:00 2001 From: STEVE MURPHY Date: Tue, 23 Aug 2022 10:42:19 -0600 Subject: [PATCH] add files for snapshot creation (#873) * add files for snapshot creation Signed-off-by: Steve Murphy * Create new topics and fix existing. Modify index.md Signed-off-by: Steve Murphy * Address Fanit's edits. Apply edits to other files as needed for consistenty. Create new files: get-snapshot.status.md and restore-snapshot.md. Signed-off-by: Steve Murphy * Add .md file extension to get-snapshot-repository file Signed-off-by: Steve Murphy * Add verify-snapshot-repository.md, renumber nav_order in other snapshot topics to place the new topic correctly. Signed-off-by: Steve Murphy * Apply Fanit's edits to restore-snapsnot.md and verify-snapshot-repository.md Signed-off-by: Steve Murphy * Address Fanit's edits. Signed-off-by: Steve Murphy * Change 'Response fields' heading level to 3. Signed-off-by: Steve Murphy Signed-off-by: Steve Murphy Co-authored-by: Steve Murphy --- .../rest-api/snapshots/create-repository.md | 88 ++++ .../rest-api/snapshots/create-snapshot.md | 134 ++++++ .../snapshots/delete-snapshot-repository.md | 41 ++ .../rest-api/snapshots/delete-snapshot.md | 43 ++ .../snapshots/get-snapshot-repository.md | 61 +++ .../rest-api/snapshots/get-snapshot-status.md | 421 ++++++++++++++++++ .../rest-api/snapshots/get-snapshot.md | 92 ++++ _opensearch/rest-api/snapshots/index.md | 13 + .../rest-api/snapshots/restore-snapshot.md | 117 +++++ .../snapshots/verify-snapshot-repository.md | 75 ++++ 10 files changed, 1085 insertions(+) create mode 100644 _opensearch/rest-api/snapshots/create-repository.md create mode 100644 _opensearch/rest-api/snapshots/create-snapshot.md create mode 100644 _opensearch/rest-api/snapshots/delete-snapshot-repository.md create mode 100644 _opensearch/rest-api/snapshots/delete-snapshot.md create mode 100644 _opensearch/rest-api/snapshots/get-snapshot-repository.md create mode 100644 _opensearch/rest-api/snapshots/get-snapshot-status.md create mode 100644 _opensearch/rest-api/snapshots/get-snapshot.md create mode 100644 _opensearch/rest-api/snapshots/index.md create mode 100644 _opensearch/rest-api/snapshots/restore-snapshot.md create mode 100644 _opensearch/rest-api/snapshots/verify-snapshot-repository.md diff --git a/_opensearch/rest-api/snapshots/create-repository.md b/_opensearch/rest-api/snapshots/create-repository.md new file mode 100644 index 00000000..afc72c22 --- /dev/null +++ b/_opensearch/rest-api/snapshots/create-repository.md @@ -0,0 +1,88 @@ +--- +layout: default +title: Register Snapshot Repository +parent: Snapshot APIs +grand_parent: REST API reference +nav_order: 1 +--- + +## Register or update snapshot repository + +Registers a new repository to store snapshots, or updates information for an existing repository. Repositories can be of two types: + +* File system (`fs`) + +* Amazon S3 bucket (`s3`) + +To learn more about repositories, see [Register repository]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#register-repository). + +### Path parameters + +Parameter | Data Type | Description +:--- | :--- | :--- +repository | String | Repostory name. | + +### Request fields + +Allowable parameters depend on the repository type. + +#### fs repository type + +Setting | Description +:--- | :--- +`location` | The shared file system for snapshots. Required. +`chunk_size` | Breaks large files into chunks during snapshot operations (e.g. `64mb`, `1gb`), which is important for cloud storage providers and far less important for shared file systems. Default is `null` (unlimited). Optional. +`compress` | Whether to compress metadata files. This setting does not affect data files, which might already be compressed, depending on your index settings. Default is `false`. Optional. +`max_restore_bytes_per_sec` | The maximum rate at which snapshots restore. Default is 40 MB per second (`40m`). Optional. +`max_snapshot_bytes_per_sec` | The maximum rate at which snapshots take. Default is 40 MB per second (`40m`). Optional. +`readonly` | Whether the repository is read-only. Useful when migrating from one cluster (`"readonly": false` when registering) to another cluster (`"readonly": true` when registering). Optional. + +#### s3 repository type + +Setting | Description +:--- | :--- +`base_path` | The path within the bucket where you want to store snapshots (e.g. `my/snapshot/directory`). Optional. If not specified, snapshots are stored in the bucket root. +`bucket` | Name of the S3 bucket. Required. +`buffer_size` | The threshold beyond which chunks (of `chunk_size`) should be broken into pieces (of `buffer_size`) and sent to S3 using a different API. Default is the smaller of two values: 100 MB or 5% of the Java heap. Valid values are between `5mb` and `5gb`. We don't recommend changing this option. +`canned_acl` | S3 has several [canned ACLs](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl) that the `repository-s3` plugin can add to objects as it creates them in S3. Default is `private`. Optional. +`chunk_size` | Breaks files into chunks during snapshot operations (e.g. `64mb`, `1gb`), which is important for cloud storage providers and far less important for shared file systems. Default is `1gb`. Optional. +`client` | When specifying client settings (e.g. `s3.client.default.access_key`), you can use a string other than `default` (e.g. `s3.client.backup-role.access_key`). If you used an alternate name, change this value to match. Default and recommended value is `default`. Optional. +`compress` | Whether to compress metadata files. This setting does not affect data files, which might already be compressed, depending on your index settings. Default is `false`. Optional. +`max_restore_bytes_per_sec` | The maximum rate at which snapshots restore. Default is 40 MB per second (`40m`). Optional. +`max_snapshot_bytes_per_sec` | The maximum rate at which snapshots take. Default is 40 MB per second (`40m`). Optional. +`readonly` | Whether the repository is read-only. Useful when migrating from one cluster (`"readonly": false` when registering) to another cluster (`"readonly": true` when registering). Optional. +`server_side_encryption` | Whether to encrypt snapshot files in the S3 bucket. This setting uses AES-256 with S3-managed keys. See [Protecting data using server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html). Default is false. Optional. +`storage_class` | Specifies the [S3 storage class](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html) for the snapshots files. Default is `standard`. Do not use the `glacier` and `deep_archive` storage classes. Optional. + +#### Sample request + +The following requests register or update a repository called `my-first-repo`. Both require a request body as described in [Request fields](#request-fields). + +* `POST _snapshot/my-first-repo/` registers a snapshot repository. + +* `PUT _snapshot/my-first-repo/` registers a snapshot repository if it does not exist; otherwise, updates information about the repository. + +The following request registers a new S3 repository called `my-opensearch-repo` in an existing bucket called `my-open-search-bucket`. By default, all snapshots are stored in the `my/snapshot/directory`. + +```json +PUT _snapshot/my-opensearch-repo +{ + "type": "s3", + "settings": { + "bucket": "my-open-search-bucket", + "base_path": "my/snapshot/directory" + } +} +``` + +#### Sample response + +Upon success, the following JSON object is returned: + +```json +{ + "acknowledged": true +} +``` +To verify that the repository was registered, use the [Get snapshot repository API]({{site.url}}{{site.baseurl}}/opensearch/rest-api/snapshots/get-snapshot-repository), passing the repository name as the `repository` path parameter. +{: .note} \ No newline at end of file diff --git a/_opensearch/rest-api/snapshots/create-snapshot.md b/_opensearch/rest-api/snapshots/create-snapshot.md new file mode 100644 index 00000000..4a5a4bed --- /dev/null +++ b/_opensearch/rest-api/snapshots/create-snapshot.md @@ -0,0 +1,134 @@ +--- +layout: default +title: Create Snapshot +parent: Snapshot APIs +grand_parent: REST API reference +nav_order: 5 +--- + +## Create snapshot + +Creates a snapshot within an existing repository. + +* To learn more about snapshots, see [Snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/index). + +* To view a list of your repositories, see [Get snapshot repository]({{site.url}}{{site.baseurl}}/opensearch//rest-api/snapshots/get-snapshot-repository). + +### Path parameters + +Parameter | Data Type | Description +:--- | :--- | :--- +repository | String | Repostory name to contain the snapshot. | +snapshot | String | Name of Snapshot to create. | + +### Query parameters + +Parameter | Data Type | Description +:--- | :--- | :--- +wait_for_completion | Boolean | Whether to wait for snapshot creation to complete before continuing. If you include this parameter, the snapshot definition is returned after completion. | + +### Request fields + +The request body is optional. + +Field | Data Type | Description +:--- | :--- | :--- +`indices` | String | The indices you want to include in the snapshot. You can use `,` to create a list of indices, `*` to specify an index pattern, and `-` to exclude certain indices. Don't put spaces between items. Default is all indices. +`ignore_unavailable` | Boolean | If an index from the `indices` list doesn't exist, whether to ignore it rather than fail the snapshot. Default is false. +`include_global_state` | Boolean | Whether to include cluster state in the snapshot. Default is true. +`partial` | Boolean | Whether to allow partial snapshots. Default is false, which fails the entire snapshot if one or more shards fails to stor + +#### Sample requests + +##### Request without a body + +The following request creates a snapshoted called `my-first-snapshot` in an S3 repository called `my-s3-repository`. A request body is not included because it is optional. + +```json +POST _snapshot/my-s3-repository/my-first-snapshot +``` + +##### Request with a body + +You can also add a request body to include or exclude certain indices or specify other settings: + +```json +PUT _snapshot/my-s3-repository/2 +{ + "indices": "opensearch-dashboards*,my-index*,-my-index-2016", + "ignore_unavailable": true, + "include_global_state": false, + "partial": false +} +```` + +#### Sample responses + +Upon success, the response content depends on whether you include the `wait_for_completion` query parameter. + +##### `wait_for_completion` not included + +```json +{ + "accepted": true +} +``` +To verify that the snapshot was created, use the [Get snapshot API]({{site.url}}{{site.baseurl}}/opensearch/rest-api/snapshots/get-snapshot), passing the snapshot name as the `snapshot` path parameter. +{: .note} + +##### `wait_for_completion` included + +The snapshot definition is returned. + +```json +{ + "snapshot" : { + "snapshot" : "5", + "uuid" : "ZRH4Zv7cSnuYev2JpLMJGw", + "version_id" : 136217927, + "version" : "2.0.1", + "indices" : [ + ".opendistro-reports-instances", + ".opensearch-observability", + ".kibana_1", + "opensearch_dashboards_sample_data_flights", + ".opensearch-notifications-config", + ".opendistro-reports-definitions", + "shakespeare" + ], + "data_streams" : [ ], + "include_global_state" : true, + "state" : "SUCCESS", + "start_time" : "2022-08-10T16:52:15.277Z", + "start_time_in_millis" : 1660150335277, + "end_time" : "2022-08-10T16:52:18.699Z", + "end_time_in_millis" : 1660150338699, + "duration_in_millis" : 3422, + "failures" : [ ], + "shards" : { + "total" : 7, + "failed" : 0, + "successful" : 7 + } + } +} +``` +#### Response fields + +| Field | Data Type | Description | +| :--- | :--- | :--- | +| snapshot | string | Snapshot name. | +| uuid | string | Snapshot's universally unique identifier (UUID). | +| version_id | int | Build ID of the Open Search version that created the snapshot. | +| version | float | Open Search version that created the snapshot. | +| indices | array | Indices in the snapshot. | +| data_streams | array | Data streams in the snapshot. | +| include_global_state | boolean | Whether the current cluster state is included in the snapshot. | +| start_time | string | Date/time when the snapshot creation process began. | +| start_time_in_millis | long | Time (in milliseconds) when the snapshot creation process began. | +| end_time | string | Date/time when the snapshot creation process ended. | +| end_time_in_millis | long | Time (in milliseconds) when the snapshot creation process ended. | +| duration_in_millis | long | Total time (in milliseconds) that the snapshot creation process lasted. | +| failures | array | Failures, if any, that occured during snapshot creation. | +| shards | object | Total number of shards created along with number of successful and failed shards. | +| state | string | Snapshot status. Possible values: `IN_PROGRESS`, `SUCCESS`, `FAILED`, `PARTIAL`. | \ No newline at end of file diff --git a/_opensearch/rest-api/snapshots/delete-snapshot-repository.md b/_opensearch/rest-api/snapshots/delete-snapshot-repository.md new file mode 100644 index 00000000..f6b5a2c3 --- /dev/null +++ b/_opensearch/rest-api/snapshots/delete-snapshot-repository.md @@ -0,0 +1,41 @@ +--- +layout: default +title: Delete Snapshot Repository +parent: Snapshot APIs +grand_parent: REST API reference +nav_order: 3 +--- + +## Delete snapshot repository configuration. + + Deletes a snapshot repository configuration. + + A repository in OpenSearch is simply a configuration that maps a repository name to a type (file system or s3 repository) along with other information depending on the type. The configuration is backed by a file system location or an s3 bucket. When you invoke the API, the physical file system or s3 bucket itself is not deleted. Only the configuration is deleted. + + To learn more about repositories, see [Register or update snapshot repository]({{site.url}}{{site.baseurl}}/opensearch/rest-api/snapshots/create-repository). + +### Path parameters + +Parameter | Data Type | Description +:--- | :--- | :--- +repository | String | Repository to delete. | + +#### Sample request + +The following request deletes the `my-opensearch-repo` repository: + +````json +DELETE _snapshot/my-opensearch-repo +```` + +#### Sample response + +Upon success, the response returns the following JSON object: + +````json +{ + "acknowledged" : true +} +```` +To verify that the repository was deleted, use the [Get snapshot repository API]({{site.url}}{{site.baseurl}}/opensearch/rest-api/snapshots/get-snapshot-repository), passing the repository name as the `repository` path parameter. +{: .note} \ No newline at end of file diff --git a/_opensearch/rest-api/snapshots/delete-snapshot.md b/_opensearch/rest-api/snapshots/delete-snapshot.md new file mode 100644 index 00000000..ac5ac653 --- /dev/null +++ b/_opensearch/rest-api/snapshots/delete-snapshot.md @@ -0,0 +1,43 @@ +--- +layout: default +title: Delete Snapshot +parent: Snapshot APIs +grand_parent: REST API reference +nav_order: 7 +--- + +## Delete snapshot + +Deletes a snapshot from a repository. + +* To learn more about snapshots, see [Snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/index). + +* To view a list of your repositories, see [cat repositories]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cat/cat-repositories). + +* To view a list of your snapshots, see [cat snapshots]({{site.url}}{{site.baseurl}}/opensearch/rest-api/cat/cat-snapshots). + +### Path parameters + +Parameter | Data Type | Description +:--- | :--- | :--- +repository | String | Repostory that contains the snapshot. | +snapshot | String | Snapshot to delete. | + +#### Sample request + +The following request deletes a snapshot called `my-first-snapshot` from the `my-opensearch-repo` repository. + +`DELETE _snapshot/my-opensearch-repo/my-first-snapshot` + +#### Sample response + +Upon success, the response returns the following JSON object: + +```json +{ + "acknowledged": true +} +``` + +To verify that the snapshot was deleted, use the [Get snapshot API]({{site.url}}{{site.baseurl}}/opensearch/rest-api/snapshots/get-snapshot), passing the snapshot name as the `snapshot` path parameter. +{: .note} diff --git a/_opensearch/rest-api/snapshots/get-snapshot-repository.md b/_opensearch/rest-api/snapshots/get-snapshot-repository.md new file mode 100644 index 00000000..0979c894 --- /dev/null +++ b/_opensearch/rest-api/snapshots/get-snapshot-repository.md @@ -0,0 +1,61 @@ +--- +layout: default +title: Get Snapshot Repository +parent: Snapshot APIs +grand_parent: REST API reference +nav_order: 2 +--- + +## Get snapshot repository. + +Retrieves information about a snapshot repository. + +To learn more about repositories, see [Register repository]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#register-repository). + +You can also get details about a snapshot during and after snapshot creation. See [Get snapshot status]({{site.url}}{{site.baseurl}}/opensearch/snapshots/get-snapshot-status). +{: .note} + +### Path parameters + +| Parameter | Data Type | Description | +| :--- | :--- | :--- | +| repository | String | A comma-separated list of snapshot repository names to retrieve. Wildcard (`*`) expressions are supported including combining wildcards with exclude patterns starting with `-`. | + +### Query parameters + +| Parameter | Data Type | Description | +:--- | :--- | :--- +| local | Boolean | Whether to get information from the local node. Optional, defaults to `false`.| +| cluster_manager_timeout | Time | Amount of time to wait for a connection to the master node. Optional, defaults to 30 seconds. | + +#### Sample request + +The following request retrieves information for the `my-opensearch-repo` repository: + +````json +GET /_snapshot/my-opensearch-repo +```` + +#### Sample response + +Upon success, the response returns repositry information. This sample is for an `s3` repository type. + +````json +{ + "my-opensearch-repo" : { + "type" : "s3", + "settings" : { + "bucket" : "my-open-search-bucket", + "base_path" : "snapshots" + } + } +} +```` + +### Response fields + +| Field | Data Type | Description | +| :--- | :--- | :--- | +| type | string | Bucket type: `fs` (file system) or `s3` (s3 bucket) | +| bucket | string | S3 bucket name. | +| base_path | string | Folder within the bucket where snapshots are stored. | \ No newline at end of file diff --git a/_opensearch/rest-api/snapshots/get-snapshot-status.md b/_opensearch/rest-api/snapshots/get-snapshot-status.md new file mode 100644 index 00000000..5cd7d5ce --- /dev/null +++ b/_opensearch/rest-api/snapshots/get-snapshot-status.md @@ -0,0 +1,421 @@ +--- +layout: default +title: Get Snapshot Status +parent: Snapshot APIs +grand_parent: REST API reference +nav_order: 8 +--- + +## Get snapshot status + +Returns details about a snapshot’s state during and after snapshot creation. + +To learn about snapshot creation, see [Create snapshot]({{site.url}}{{site.baseurl}}/opensearch/rest-api/snapshots/create-snapshot). + +If you use the security plugin, you must have the `monitor_snapshot`, `create_snapshot`, or `manage cluster` privileges. +{: .note} + +### Path parameters + +Path parameters are optional. + +| Parameter | Data Type | Description | +:--- | :--- | :--- +| repository | String | Repository containing the snapshot. | +| snapshot | String | Snapshot to return. | + +Three request variants provide flexibility: + +* `GET _snapshot/_status` returns the status of all currently running snapshots in all repositories. + +* `GET _snapshot//_status` returns the status of only currently running snapshots in the specified repository. This is the preferred variant. + +* `GET _snapshot///_status` returns the status of all snapshots in the specified repository whether they are running or not. + +Using the API to return state for other than currently running snapshots can be very costly for (1) machine machine resources and (2) processing time if running in the cloud. For each snapshot, each request causes file reads from all a snapshot's shards. +{: .warning} + +### Request fields + +| Field | Data Type | Description | +:--- | :--- | :--- +| ignore_unavailable | Boolean | How to handles requests for unavailable snapshots. If `false`, the request returns an error for unavailable snapshots. If `true`, the request ignores unavailable snapshots, such as those that are corrupted or temporarily cannot be returned. Defaults to `false`.| + +#### Sample request + +The following request returns the status of `my-first-snapshot` in the `my-opensearch-repo` repository. Unavailable snapshots are ignored. + +````json +GET _snapshot/my-opensearch-repo/my-first-snapshot/_status +{ + "ignore_unavailable": true +} +```` + +#### Sample response + +The example that follows corresponds to the request above in the [Sample request](#sample-request) section. + +The `GET _snapshot/my-opensearch-repo/my-first-snapshot/_status` request returns the following fields: + +````json +{ + "snapshots" : [ + { + "snapshot" : "my-first-snapshot", + "repository" : "my-opensearch-repo", + "uuid" : "dCK4Qth-TymRQ7Tu7Iga0g", + "state" : "SUCCESS", + "include_global_state" : true, + "shards_stats" : { + "initializing" : 0, + "started" : 0, + "finalizing" : 0, + "done" : 7, + "failed" : 0, + "total" : 7 + }, + "stats" : { + "incremental" : { + "file_count" : 31, + "size_in_bytes" : 24488927 + }, + "total" : { + "file_count" : 31, + "size_in_bytes" : 24488927 + }, + "start_time_in_millis" : 1660666841667, + "time_in_millis" : 14054 + }, + "indices" : { + ".opensearch-observability" : { + "shards_stats" : { + "initializing" : 0, + "started" : 0, + "finalizing" : 0, + "done" : 1, + "failed" : 0, + "total" : 1 + }, + "stats" : { + "incremental" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "total" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "start_time_in_millis" : 1660666841868, + "time_in_millis" : 201 + }, + "shards" : { + "0" : { + "stage" : "DONE", + "stats" : { + "incremental" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "total" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "start_time_in_millis" : 1660666841868, + "time_in_millis" : 201 + } + } + } + }, + "shakespeare" : { + "shards_stats" : { + "initializing" : 0, + "started" : 0, + "finalizing" : 0, + "done" : 1, + "failed" : 0, + "total" : 1 + }, + "stats" : { + "incremental" : { + "file_count" : 4, + "size_in_bytes" : 18310117 + }, + "total" : { + "file_count" : 4, + "size_in_bytes" : 18310117 + }, + "start_time_in_millis" : 1660666842470, + "time_in_millis" : 13050 + }, + "shards" : { + "0" : { + "stage" : "DONE", + "stats" : { + "incremental" : { + "file_count" : 4, + "size_in_bytes" : 18310117 + }, + "total" : { + "file_count" : 4, + "size_in_bytes" : 18310117 + }, + "start_time_in_millis" : 1660666842470, + "time_in_millis" : 13050 + } + } + } + }, + "opensearch_dashboards_sample_data_flights" : { + "shards_stats" : { + "initializing" : 0, + "started" : 0, + "finalizing" : 0, + "done" : 1, + "failed" : 0, + "total" : 1 + }, + "stats" : { + "incremental" : { + "file_count" : 10, + "size_in_bytes" : 6132245 + }, + "total" : { + "file_count" : 10, + "size_in_bytes" : 6132245 + }, + "start_time_in_millis" : 1660666843476, + "time_in_millis" : 6221 + }, + "shards" : { + "0" : { + "stage" : "DONE", + "stats" : { + "incremental" : { + "file_count" : 10, + "size_in_bytes" : 6132245 + }, + "total" : { + "file_count" : 10, + "size_in_bytes" : 6132245 + }, + "start_time_in_millis" : 1660666843476, + "time_in_millis" : 6221 + } + } + } + }, + ".opendistro-reports-definitions" : { + "shards_stats" : { + "initializing" : 0, + "started" : 0, + "finalizing" : 0, + "done" : 1, + "failed" : 0, + "total" : 1 + }, + "stats" : { + "incremental" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "total" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "start_time_in_millis" : 1660666843076, + "time_in_millis" : 200 + }, + "shards" : { + "0" : { + "stage" : "DONE", + "stats" : { + "incremental" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "total" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "start_time_in_millis" : 1660666843076, + "time_in_millis" : 200 + } + } + } + }, + ".opendistro-reports-instances" : { + "shards_stats" : { + "initializing" : 0, + "started" : 0, + "finalizing" : 0, + "done" : 1, + "failed" : 0, + "total" : 1 + }, + "stats" : { + "incremental" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "total" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "start_time_in_millis" : 1660666841667, + "time_in_millis" : 201 + }, + "shards" : { + "0" : { + "stage" : "DONE", + "stats" : { + "incremental" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "total" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "start_time_in_millis" : 1660666841667, + "time_in_millis" : 201 + } + } + } + }, + ".kibana_1" : { + "shards_stats" : { + "initializing" : 0, + "started" : 0, + "finalizing" : 0, + "done" : 1, + "failed" : 0, + "total" : 1 + }, + "stats" : { + "incremental" : { + "file_count" : 13, + "size_in_bytes" : 45733 + }, + "total" : { + "file_count" : 13, + "size_in_bytes" : 45733 + }, + "start_time_in_millis" : 1660666842673, + "time_in_millis" : 2007 + }, + "shards" : { + "0" : { + "stage" : "DONE", + "stats" : { + "incremental" : { + "file_count" : 13, + "size_in_bytes" : 45733 + }, + "total" : { + "file_count" : 13, + "size_in_bytes" : 45733 + }, + "start_time_in_millis" : 1660666842673, + "time_in_millis" : 2007 + } + } + } + }, + ".opensearch-notifications-config" : { + "shards_stats" : { + "initializing" : 0, + "started" : 0, + "finalizing" : 0, + "done" : 1, + "failed" : 0, + "total" : 1 + }, + "stats" : { + "incremental" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "total" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "start_time_in_millis" : 1660666842270, + "time_in_millis" : 200 + }, + "shards" : { + "0" : { + "stage" : "DONE", + "stats" : { + "incremental" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "total" : { + "file_count" : 1, + "size_in_bytes" : 208 + }, + "start_time_in_millis" : 1660666842270, + "time_in_millis" : 200 + } + } + } + } + } + } + ] +} +```` + +### Response fields + +| Field | Data Type | Description | +:--- | :--- | :--- +| repository | String | Name of repository that contains the snapshot. | +| snapshot | String | Snapshot name. | +| uuid | String | Snapshot Universally unique identifier (UUID). | +| state | String | Snapshot's current status. See [Snapshot states](#snapshot-states). | +| include_global_state | Boolean | Whether the current cluster state is included in the snapshot. | +| shards_stats | Object | Snapshot's shard counts. See [Shard stats](#shard-stats). | +| stats | Object | Details of files included in the snapshot. `file_count`: number of files. `size_in_bytes`: total of all fie sizes. See [Snapshot file stats](#snapshot-file-stats). | +| index | list of Objects | List of objects that contain information about the indices in the snapshot. See [Index objects](#index-objects).| + +##### Snapshot states + +| State | Description | +:--- | :--- | +| FAILED | The snapshot terminated in an error and no data was stored. | +| IN_PROGRESS | The snapshot is currently running. | +| PARTIAL | The global cluster state was stored, but data from at least one shard was not stored. The `failures` property of the [Create snapshot]({{site.url}}{{site.baseurl}}/opensearch/rest-api/snapshots/create-snapshot) response contains additional details. | +| SUCCESS | The snapshot finished and all shards were stored successfully. | + +##### Shard stats + +All property values are Integers. + +| Property | Description | +:--- | :--- | +| initializing | Number of shards that are still initializing. | +| started | Number of shards that have started but not are not finalized. | +| finalizing | Number of shards that are finalizing but are not done. | +| done | Number of shards that initialized, started, and finalized successfully. | +| failed | Number of shards that failed to be included in the snapshot. | +| total | Total number of shards included in the snapshot. | + +##### Snapshot file stats + +| Property | Type | Description | +:--- | :--- | :--- | +| incremental | Object | Number and size of files that still need to be copied during snapshot creation. For completed snapshots, `incremental` provides the number and size of files that were not already in the repository and were copied as part of the incremental snapshot. | +| processed | Object | Number and size of files already uploaded to the snapshot. The processed `file_count` and `size_in_bytes` are incremented in stats after a file is uploaded. | +| total | Object | Total number and size of files that are referenced by the snapshot. | +| start_time_in_millis | Long | Time (in milliseconds) when snapshot creation began. | +| time_in_millis | Long | Total time (in milliseconds) that the snapshot took to complete. | + +##### Index objects + +| Property | Type | Description | +:--- | :--- | :--- | +| shards_stats | Object | See [Shard stats](#shard-stats). | +| stats | Object | See [Snapshot file stats](#snapshot-file-stats). | +| shards | list of Objects | List of objects containing information about the shards that include the snapshot. Properies of the shards are listed below in bold text.

**stage**: Current state of shards in the snapshot. Shard states are:

* DONE: Number of shards in the snapshot that were successfully stored in the repository.

* FAILURE: Number of shards in the snapshot that were not successfully stored in the repository.

* FINALIZE: Number of shards in the snapshot that are in the finalizing stage of being stored in the repository.

* INIT: Number of shards in the snapshot that are in the initializing stage of being stored in the repository.

* STARTED: Number of shards in the snapshot that are in the started stage of being stored in the repository.

**stats**: See [Snapshot file stats](#snapshot-file-stats).

**total**: Total number and size of files referenced by the snapshot.

**start_time_in_millis**: Time (in milliseconds) when snapshot creation began.

**time_in_millis**: Total time (in milliseconds) that the snapshot took to complete. | \ No newline at end of file diff --git a/_opensearch/rest-api/snapshots/get-snapshot.md b/_opensearch/rest-api/snapshots/get-snapshot.md new file mode 100644 index 00000000..ec7e1e20 --- /dev/null +++ b/_opensearch/rest-api/snapshots/get-snapshot.md @@ -0,0 +1,92 @@ +--- +layout: default +title: Get Snapshot +parent: Snapshot APIs +grand_parent: REST API reference +nav_order: 6 +--- + +## Get snapshot. + +Retrieves information about a snapshot. + +### Path parameters + +| Parameter | Data Type | Description | +| :--- | :--- | :--- | +| repository | String | The repository that contains the snapshot to retrieve. | +| snapshot | String | Snapshot to retrieve. + +### Query parameters + +| Parameter | Data Type | Description | +:--- | :--- | :--- +| verbose | Boolean | Whether to show all, or just basic snapshot information. If `true`, returns all information. If `false`, omits information like start/end times, failures, and shards. Optional, defaults to `true`.| +| ignore_unavailable | Boolean | How to handle snapshots that are unavailable (corrupted or otherwise temporarily can't be returned). If `true` and the snapshot is unavailable, the request does not return the snapshot. If `false` and the snapshot is unavailable, the request returns an error. Optional, defaults to `false`.| + +#### Sample request + +The following request retrieves information for the `my-first-snapshot` located in the `my-opensearch-repo` repository: + +````json +GET _snapshot/my-opensearch-repo/my-first-snapshot +```` + +#### Sample response + +Upon success, the response returns snapshot information: + +````json +{ + "snapshots" : [ + { + "snapshot" : "my-first-snapshot", + "uuid" : "3P7Qa-M8RU6l16Od5n7Lxg", + "version_id" : 136217927, + "version" : "2.0.1", + "indices" : [ + ".opensearch-observability", + ".opendistro-reports-instances", + ".opensearch-notifications-config", + "shakespeare", + ".opendistro-reports-definitions", + "opensearch_dashboards_sample_data_flights", + ".kibana_1" + ], + "data_streams" : [ ], + "include_global_state" : true, + "state" : "SUCCESS", + "start_time" : "2022-08-11T20:30:00.399Z", + "start_time_in_millis" : 1660249800399, + "end_time" : "2022-08-11T20:30:14.851Z", + "end_time_in_millis" : 1660249814851, + "duration_in_millis" : 14452, + "failures" : [ ], + "shards" : { + "total" : 7, + "failed" : 0, + "successful" : 7 + } + } + ] +} +```` +### Response fields + +| Field | Data Type | Description | +| :--- | :--- | :--- | +| snapshot | string | Snapshot name. | +| uuid | string | Snapshot's universally unique identifier (UUID). | +| version_id | int | Build ID of the Open Search version that created the snapshot. | +| version | float | Open Search version that created the snapshot. | +| indices | array | Indices in the snapshot. | +| data_streams | array | Data streams in the snapshot. | +| include_global_state | boolean | Whether the current cluster state is included in the snapshot. | +| start_time | string | Date/time when the snapshot creation process began. | +| start_time_in_millis | long | Time (in milliseconds) when the snapshot creation process began. | +| end_time | string | Date/time when the snapshot creation process ended. | +| end_time_in_millis | long | Time (in milliseconds) when the snapshot creation process ended. | +| duration_in_millis | long | Total time (in milliseconds) that the snapshot creation process lasted. | +| failures | array | Failures, if any, that occured during snapshot creation. | +| shards | object | Total number of shards created along with number of successful and failed shards. | +| state | string | Snapshot status. Possible values: `IN_PROGRESS`, `SUCCESS`, `FAILED`, `PARTIAL`. | \ No newline at end of file diff --git a/_opensearch/rest-api/snapshots/index.md b/_opensearch/rest-api/snapshots/index.md new file mode 100644 index 00000000..0b167bca --- /dev/null +++ b/_opensearch/rest-api/snapshots/index.md @@ -0,0 +1,13 @@ +--- +layout: default +title: Snapshot APIs +parent: REST API reference +has_children: true +nav_order: 6 +redirect_from: + - /opensearch/rest-api/document-apis/ +--- + +# Snapshot APIs + +The snapshot APIs allow you to manage snapshots and snapshot repositories. \ No newline at end of file diff --git a/_opensearch/rest-api/snapshots/restore-snapshot.md b/_opensearch/rest-api/snapshots/restore-snapshot.md new file mode 100644 index 00000000..a09e8b99 --- /dev/null +++ b/_opensearch/rest-api/snapshots/restore-snapshot.md @@ -0,0 +1,117 @@ +--- +layout: default +title: Restore Snapshot +parent: Snapshot APIs +grand_parent: REST API reference +nav_order: 9 +--- + +## Restore Snapshot + +Restores a snapshot of a cluster or specified data streams and indices. + +* For information about indices and clusters, see [Introduction to OpenSearch]({{site.url}}{{site.baseurl}}/opensearch/index). + +* For information about data streams, see [Data streams]({{site.url}}{{site.baseurl}}/opensearch/data-streams). + +If open indices with the same name that you want to restore already exist in the cluster, you must close, delete, or rename the indices. See [Sample Request](#sample-request-1) for information about renaming an index. See [Close index]({{site.url}}{{site.baseurl}}/opensearch/rest-api/index-apis/close-index) for information about closing an index. +{: .note} + +### Path parameters + +| Parameter | Data Type | Description | +:--- | :--- | :--- +repository | String | Repository containing the snapshot to restore. | +| snapshot | String | Snapshot to restore. | + +### Query parameters + +Parameter | Data Type | Description +:--- | :--- | :--- +wait_for_completion | Boolean | Whether to wait for snapshot restoration to complete before continuing. | + +### Request fields + +All request body parameters are optional. + +| Parameter | Data Type | Description | +:--- | :--- | :--- +| ignore_unavailable | Boolean | How to handle data streams or indices that are missing or closed. If `false`, the request returns an error for any data stream or index that is missing or closed. If `true`, the request ignores data streams and indices in indices that are missing or closed. Defaults to `false`. | +| ignore_index_settings | Boolean | A comma-delimited list of index settings that you don't want to restore from a snapshot. | +| include_aliases | Boolean | How to handle index aliases from the original snapshot. If `true`, index aliases from the original snapshot are restored. If `false`, aliases along with associated indices are not restored. Defaults to `true`. | +| include_global_state | Boolean | Whether to restore the current cluster state1. If `false`, the cluster state is not restored. If true, the current cluster state is restored. Defaults to `false`.| +| index_settings | String | A comma-delimited list of settings to add or change in all restored indices. Use this parameter to override index settings during snapshot restoration. For data streams, these index settings are applied to the restored backing indices. | +| indices | String | A comma-delimited list of data streams and indices to restore from the snapshot. Multi-index syntax is supported. By default, a restore operation includes all data streams and indices in the snapshot. If this argument is provided, the restore operation only includes the data streams and indices that you specify. | +| partial | Boolean | How the restore operation will behave if indices in the snapshot do not have all primary shards available. If `false`, the entire restore operation fails if any indices in the snapshot do not have all primary shards available.

If `true`, allows the restoration of a partial snapshot of indices with unavailable shards. Only shards that were successfully included in the snapshot are restored. All missing shards are recreated as empty. By default, the entire restore operation fails if one or more indices included in the snapshot do not have all primary shards available. To change this behavior, set `partial` to `true`. Defaults to `false`. | +| rename_pattern | String | The pattern to apply to restored data streams and indices. Data streams and indices matching the rename pattern will be renamed according to `rename_replacement`.

The rename pattern is applied as defined by the regular expression that supports referencing the original text.

The request fails if two or more data streams or indices are renamed into the same name. If you rename a restored data stream, its backing indices are also renamed. For example, if you rename the logs data stream to `recovered-logs`, the backing index `.ds-logs-1` is renamed to `.ds-recovered-logs-1`.

If you rename a restored stream, ensure an index template matches the new stream name. If there are no matching index template names, the stream cannot roll over and new backing indices are not created.| +| rename_replacement | String | The rename replacement string. See `rename_pattern` for more information.| +| wait_for_completion | Boolean | Whether to return a response after the restore operation has completed. If `false`, the request returns a response when the restore operation initializes. If `true`, the request returns a response when the restore operation completes. Defaults to `false`. | + +1The cluster state includes: +* Persistent cluster settings +* Index templates +* Legacy index templates +* Ingest pipelines +* Index lifecycle policies + +#### Sample request + +The following request restores the `opendistro-reports-definitions` index from `my-first-snapshot`. The `rename_pattern` and `rename_replacement` combination causes the index to be renamed to `opendistro-reports-definitions_restored` because duplicate open index names in a cluster are not allowed. + +````json +POST /_snapshot/my-opensearch-repo/my-first-snapshot/_restore +{ + "indices": "opendistro-reports-definitions", + "ignore_unavailable": true, + "include_global_state": false, + "rename_pattern": "(.+)", + "rename_replacement": "$1_restored", + "include_aliases": false +} +```` + +#### Sample Response + +Upon success, the response returns the following JSON object: + +````json +{ + "snapshot" : { + "snapshot" : "my-first-snapshot", + "indices" : [ ], + "shards" : { + "total" : 0, + "failed" : 0, + "successful" : 0 + } + } +} +```` +Except for the snapshot name, all properties are empty or `0`. This is because any changes made to the volume after the snapshot was generated are lost. However, if you invoke the [Get snapshot]({{site.url}}{{site.baseurl}}/opensearch/rest-api/snapshots/get-snapshot) API to examine the snapshot, a fully populated snapshot object is returned. + +### Response fields + +| Field | Data Type | Description | +| :--- | :--- | :--- | +| snapshot | string | Snapshot name. | +| indices | array | Indices in the snapshot. | +| shards | object | Total number of shards created along with number of successful and failed shards. | + +If open indices in a snapshot already exist in a cluster, and you don't delete, close, or rename them, the API returns an error like the following: +{: .note} + +````json +{ + "error" : { + "root_cause" : [ + { + "type" : "snapshot_restore_exception", + "reason" : "[my-opensearch-repo:my-first-snapshot/dCK4Qth-TymRQ7Tu7Iga0g] cannot restore index [.opendistro-reports-definitions] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name" + } + ], + "type" : "snapshot_restore_exception", + "reason" : "[my-opensearch-repo:my-first-snapshot/dCK4Qth-TymRQ7Tu7Iga0g] cannot restore index [.opendistro-reports-definitions] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name" + }, + "status" : 500 +} +```` \ No newline at end of file diff --git a/_opensearch/rest-api/snapshots/verify-snapshot-repository.md b/_opensearch/rest-api/snapshots/verify-snapshot-repository.md new file mode 100644 index 00000000..8e7e778f --- /dev/null +++ b/_opensearch/rest-api/snapshots/verify-snapshot-repository.md @@ -0,0 +1,75 @@ +--- +layout: default +title: Verify Snaphot Repository +parent: Snapshot APIs +grand_parent: REST API reference +nav_order: 4 +--- + +## Verify snapshot repository + +Verifies that a snapshot repository is functional. Verifies the repository on each node in a cluster. + +If verification is successful, the verify snapshot repository API returns a list of nodes connected to the snapshot repository. If verification failed, the API returns an error. + +If you use the security plugin, you must have the `manage cluster` privilege. +{: .note} + +### Path parameters + +Path parameters are optional. + +| Parameter | Data Type | Description | +:--- | :--- | :--- +| repository | String | Name of repository to verify. | + +### Query parameters + +| Parameter | Data Type | Description | +:--- | :--- | :--- +| cluster_manager_timeout | Time | Amount of time to wait for a connection to the master node. Optional, defaults to `30s`. | +| timeout | Time | The period of time to wait for a response. If a response is not received before the timeout value, the request fails and returns an error. Defaults to `30s`. | + +#### Sample request + +The following request verifies that the my-opensearch-repo is functional: + +````json +POST /_snapshot/my-opensearch-repo/_verify?timeout=0s&cluster_manager_timeout=50s +```` + +#### Sample response + +The example that follows corresponds to the request above in the [Sample request](#sample-request) section. + +The `POST /_snapshot/my-opensearch-repo/_verify?timeout=0s&cluster_manager_timeout=50s` request returns the following fields: + +````json +{ + "nodes" : { + "by1kztwTRoeCyg4iGU5Y8A" : { + "name" : "opensearch-node1" + } + } +} +```` + +In the preceding sample, one node is connected to the snapshot repository. If more were connected, you would see them in the response. Example: + +````json +{ + "nodes" : { + "lcfL6jv2jo6sMEtp4idMvg" : { + "name" : "node-1" + }, + "rEPtFT/B+cuuOHnQn0jy4s" : { + "name" : "node-2" + } +} +```` + +### Response fields + +| Field | Data Type | Description | +:--- | :--- | :--- +| nodes | Object | A list (not an array) of nodes connected to the snapshot repository. Each node itself is a property where the node ID is the key and the name has an ID (Object) and a name (String). | \ No newline at end of file