diff --git a/_im-plugin/ism/policies.md b/_im-plugin/ism/policies.md index c2a2ccda..11aa6bec 100644 --- a/_im-plugin/ism/policies.md +++ b/_im-plugin/ism/policies.md @@ -374,7 +374,7 @@ Parameter | Description | Type ### snapshot -Backup your cluster’s indexes and state. For more information about snapshots, see [Take and restore snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshot-restore/). +Back up your cluster’s indexes and state. For more information about snapshots, see [Take and restore snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore). The `snapshot` operation has the following parameters: diff --git a/_monitoring-plugins/alerting/settings.md b/_monitoring-plugins/alerting/settings.md index 64beebd2..b4a42725 100644 --- a/_monitoring-plugins/alerting/settings.md +++ b/_monitoring-plugins/alerting/settings.md @@ -16,7 +16,7 @@ Index | Purpose :--- | :--- `.opendistro-alerting-alerts` | Stores ongoing alerts. `.opendistro-alerting-alert-history-` | Stores a history of completed alerts. -`.opendistro-alerting-config` | Stores monitors, triggers, and destinations. [Take a snapshot]({{site.url}}{{site.baseurl}}/opensearch/snapshot-restore) of this index to back up your alerting configuration. +`.opendistro-alerting-config` | Stores monitors, triggers, and destinations. [Take a snapshot]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore) of this index to back up your alerting configuration. `.opendistro-alerting-alert-history-write` (alias) | Provides a consistent URI for the `.opendistro-alerting-alert-history-` index. All alerting indices are hidden by default. For a summary, make the following request: diff --git a/_opensearch/snapshots/index.md b/_opensearch/snapshots/index.md new file mode 100644 index 00000000..192f6f02 --- /dev/null +++ b/_opensearch/snapshots/index.md @@ -0,0 +1,27 @@ +--- +layout: default +title: Snapshots +nav_order: 65 +has_children: true +redirect_from: /opensearch/snapshots/ +has_toc: false +--- + +# Snapshots + +Snapshots are backups of a cluster's indexes and state. State includes cluster settings, node information, index metadata (mappings, settings, templates, etc.), and shard allocation. + +Snapshots have two main uses: + +- **Recovering from failure** + + For example, if cluster health goes red, you might restore the red indexes from a snapshot. + +- **Migrating from one cluster to another** + + For example, if you're moving from a proof-of-concept to a production cluster, you might take a snapshot of the former and restore it on the latter. + + +You can take and restore snapshots using the [snapshot API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore). + +If you need to automate taking snapshots, you can use the [Snapshot Management]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-management) feature. diff --git a/_opensearch/snapshots/sm-api.md b/_opensearch/snapshots/sm-api.md new file mode 100644 index 00000000..563e7bbf --- /dev/null +++ b/_opensearch/snapshots/sm-api.md @@ -0,0 +1,460 @@ +--- +layout: default +title: Snapshot Management API +parent: Snapshots +nav_order: 30 +has_children: false +--- + +# Snapshot Management API + +Use the [Snapshot Management (SM)]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#take-snapshots) API to automate [taking snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#take-snapshots). + +--- + +#### Table of contents +- TOC +{:toc} + + +--- + +## Create or update a policy +Introduced 2.1 +{: .label .label-purple } + +Creates or updates an SM policy. + +#### Request + +Create: + +```json +POST _plugins/_sm/policies/ +``` + +Update: + +```json +PUT _plugins/_sm/policies/?if_seq_no=0&if_primary_term=1 +``` + +You must provide the `seq_no` and `primary_term` parameters for an update request. + +### Example + +```json +POST _plugins/_sm/policies/daily-policy +{ + "description": "Daily snapshot policy", + "creation": { + "schedule": { + "cron": { + "expression": "0 8 * * *", + "timezone": "UTC" + } + }, + "time_limit": "1h" + }, + "deletion": { + "schedule": { + "cron": { + "expression": "0 1 * * *", + "timezone": "America/Los_Angeles" + } + }, + "condition": { + "max_age": "7d", + "max_count": 21, + "min_count": 7 + }, + "time_limit": "1h" + }, + "snapshot_config": { + "date_format": "yyyy-MM-dd-HH:mm", + "timezone": "America/Los_Angeles", + "indices": "*", + "repository": "s3-repo", + "ignore_unavailable": "true", + "include_global_state": "false", + "partial": "true", + "metadata": { + "any_key": "any_value" + } + }, + "notification": { + "channel": { + "id": "NC3OpoEBzEoHMX183R3f" + }, + "conditions": { + "creation": true, + "deletion": false, + "failure": false, + "time_limit_exceeded": false + } + } +} +``` + +### Response + +```json +{ + "_id" : "daily-policy-sm-policy", + "_version" : 5, + "_seq_no" : 54983, + "_primary_term" : 21, + "sm_policy" : { + "name" : "daily-policy", + "description" : "Daily snapshot policy", + "schema_version" : 15, + "creation" : { + "schedule" : { + "cron" : { + "expression" : "0 8 * * *", + "timezone" : "UTC" + } + }, + "time_limit" : "1h" + }, + "deletion" : { + "schedule" : { + "cron" : { + "expression" : "0 1 * * *", + "timezone" : "America/Los_Angeles" + } + }, + "condition" : { + "max_age" : "7d", + "min_count" : 7, + "max_count" : 21 + }, + "time_limit" : "1h" + }, + "snapshot_config" : { + "indices" : "*", + "metadata" : { + "any_key" : "any_value" + }, + "ignore_unavailable" : "true", + "timezone" : "America/Los_Angeles", + "include_global_state" : "false", + "date_format" : "yyyy-MM-dd-HH:mm", + "repository" : "s3-repo", + "partial" : "true" + }, + "schedule" : { + "interval" : { + "start_time" : 1656425122909, + "period" : 1, + "unit" : "Minutes" + } + }, + "enabled" : true, + "last_updated_time" : 1656425122909, + "enabled_time" : 1656425122909, + "notification" : { + "channel" : { + "id" : "NC3OpoEBzEoHMX183R3f" + }, + "conditions" : { + "creation" : true, + "deletion" : false, + "failure" : false, + "time_limit_exceeded" : false + } + } + } +} +``` + +### Parameters + +You can specify the following parameters to create/update an SM policy. + +Parameter | Type | Description +:--- | :--- |:--- |:--- | +`description` | String | The description of the SM policy. Optional. +`enabled` | Boolean | Should this SM policy be enabled at creation? Optional. +`snapshot_config` | Object | The configuration options for snapshot creation. Required. +`snapshot_config`
    `date_format` | String | Snapshot names have the format `--`. `date_format` specifies the format for the date in the snapshot name. Supports all date formats supported by OpenSearch. Optional. Default is "yyyy-MM-dd'T'HH:mm:ss". +`snapshot_config`
    `date_format_timezone` | String | Snapshot names have the format `--`. `date_format_timezone` specifies the time zone for the date in the snapshot name. Optional. Default is UTC. +`snapshot_config`
    `indices` | String | The names of the indexes in the snapshot. Multiple index names are separated by `,`. Supports wildcards (`*`). Optional. Default is `*` (all indexes). +`snapshot_config`
    `repository` | String | The repository in which to store snapshots. Required. +`snapshot_config`
    `ignore_unavailable` | Boolean | Do you want to ignore unavailable indexes? Optional. Default is `false`. +`snapshot_config`
    `include_global_state` | Boolean | Do you want to include cluster state? Optional. Default is `true` because of [Security plugin considerations]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore/#security-plugin-considerations). +`snapshot_config`
    `partial` | Boolean | Do you want to allow partial snapshots? Optional. Default is `false`. +`snapshot_config`
    `metadata` | Object | Metadata in the form of key/value pairs. Optional. +`creation` | Object | Configuration for snapshot creation. Required. +`creation`
    `schedule` | String | The cron schedule used to create snapshots. Required. +`creation`
    `time_limit` | String | Sets the maximum time to wait for snapshot creation to finish. If time_limit is longer than the scheduled time interval for taking snapshots, no scheduled snapshots are taken until time_limit elapses. For example, if time_limit is set to 35 minutes and snapshots are taken every 30 minutes starting at midnight, the snapshots at 00:00 and 01:00 are taken, but the snapshot at 00:30 is skipped. Optional. +`deletion` | Object | Configuration for snapshot deletion. Optional. Default is to retain all snapshots. +`deletion`
    `schedule` | String | The cron schedule used to delete snapshots. Optional. Default is to use `creation.schedule`, which is required. +`deletion`
    `time_limit` | String | Sets the maximum time to wait for snapshot deletion to finish. Optional. +`deletion`
    `delete_condition` | Object | Conditions for snapshot deletion. Optional. +`deletion`
    `delete_condition`
        `max_count` | Integer | The maximum number of snapshots to be retained. Optional. +`deletion`
    `delete_condition`
        `max_age` | String | The maximum time a snapshot is retained. Optional. +`deletion`
    `delete_condition`
        `min_count` | Integer | The minimum number of snapshots to be retained. Optional. Default is one. +`notification` | Object | Defines notifications for SM events. Optional. +`notification`
    `channel` | Object | Defines a channel for notifications. You must [create and configure a notification channel]({{site.url}}{{site.baseurl}}/notifications-plugin/api) before setting up SM notifications. Required. +`notification`
    `channel`
        `id` | String | The channel ID of the channel used for notifications. To get the channel IDs of all created channels, use `GET _plugins/_notifications/configs`. Required. +`notification`
    `conditions` | Object | SM events you want to be notified about. Set the ones you are interested in to `true`. +`notification`
    `conditions`
        `creation` | Boolean | Do you want notifications about snapshot creation? Optional. Default is `true`. +`notification`
    `conditions`
        `deletion` | Boolean | Do you want notifications about snapshot deletion? Optional. Default is `false`. +`notification`
    `conditions`
        `failure` | Boolean | Do you want notifications about creation or deletion failure? Optional. Default is `false`. +`notification`
    `conditions`
        `time_limit_exceeded` | Boolean | Do you want notifications when snapshot operations take longer than time_limit? Optional. Default is `false`. + +## Get policies +Introduced 2.1 +{: .label .label-purple } + +Gets SM policies. + +#### Request + +Get all SM policies: + +```json +GET _plugins/_sm/policies +``` +You can use a [query string]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/full-text/#query-string) and specify pagination, the field to be sorted by, and sort order: + +```json +GET _plugins/_sm/policies?from=0&size=20&sortField=sm_policy.name&sortOrder=desc&queryString=* +``` + +Get a specific SM policy: + +``` +GET _plugins/_sm/policies/ +``` + +### Example + +```json +GET _plugins/_sm/policies/daily-policy +``` + +### Response + +```json +{ + "_id" : "daily-policy-sm-policy", + "_version" : 6, + "_seq_no" : 44696, + "_primary_term" : 19, + "sm_policy" : { + "name" : "daily-policy", + "description" : "Daily snapshot policy", + "schema_version" : 15, + "creation" : { + "schedule" : { + "cron" : { + "expression" : "0 8 * * *", + "timezone" : "UTC" + } + }, + "time_limit" : "1h" + }, + "deletion" : { + "schedule" : { + "cron" : { + "expression" : "0 1 * * *", + "timezone" : "America/Los_Angeles" + } + }, + "condition" : { + "max_age" : "7d", + "min_count" : 7, + "max_count" : 21 + }, + "time_limit" : "1h" + }, + "snapshot_config" : { + "metadata" : { + "any_key" : "any_value" + }, + "ignore_unavailable" : "true", + "include_global_state" : "false", + "date_format" : "yyyy-MM-dd-HH:mm", + "repository" : "s3-repo", + "partial" : "true" + }, + "schedule" : { + "interval" : { + "start_time" : 1656341042874, + "period" : 1, + "unit" : "Minutes" + } + }, + "enabled" : true, + "last_updated_time" : 1656341042874, + "enabled_time" : 1656341042874 + } +} +``` + +## Explain +Introduced 2.1 +{: .label .label-purple } + +Provides the enabled/disabled status and the metadata for all policies specified. Multiple policy names are separated with `,`. You can also specify desired policies with a wildcard pattern. + +SM State Machine + +SM uses a state machine for snapshot creation and deletion. The image on the left shows one execution period of the creation workflow, from the CREATION_START state to the CREATION_FINISHED state. Deletion workflow follows the same pattern as creation workflow. + +The creation workflow starts in the CREATION_START state and continuously checks if the conditions in the creation cron schedule are met. After the conditions are met, the creation workflow switches to the CREATION_CONDITION_MET state and continues to the CREATING state. The CREATING state calls the create snapshot API asynchronously and then waits for snapshot creation to end in the CREATION_FINISHED state. Once snapshot creation ends, the creation workflow goes back to the CREATION_START state, and the cycle continues. The `current_state` field of `metadata.creation` and `metadata.deletion` returns the current state of the state machine. + +#### Request + +```json +GET _plugins/_sm/policies//_explain +``` + +### Example + +```json +GET _plugins/_sm/policies/daily*/_explain +``` + +### Response + +```json +{ + "policies" : [ + { + "name" : "daily-policy", + "creation" : { + "current_state" : "CREATION_START", + "trigger" : { + "time" : 1656403200000 + } + }, + "deletion" : { + "current_state" : "DELETION_START", + "trigger" : { + "time" : 1656403200000 + } + }, + "policy_seq_no" : 44696, + "policy_primary_term" : 19, + "enabled" : true + } + ] +} +``` + +The following table lists all fields for each policy in the response. + +Field | Description +:--- |:--- +`name` | The name of the SM policy. +`creation` | Information about the latest creation operation. See subfields below. +`deletion` | Information about the latest deletion operation. See subfields below. +`policy_seq_no`
`policy_primary_term` | The version of the SM policy. +`enabled` | Is the policy running? + +The following table lists all fields in the `creation` and `deletion` objects of each policy. + +Field | Description +:--- |:--- +`current_state` | The current state of the state machine that runs snapshot creation/deletion as described above. +`trigger.time` | The next creation/deletion execution time in milliseconds since the epoch. +`latest_execution` | Describes the latest creation/deletion execution. +`latest_execution.status` | The execution status of the latest creation/deletion. Possible values are:
`IN_PROGRESS`: Snapshot creation/deletion has started.
`SUCCESS`: Snapshot creation/deletion has finished successfully.
`RETRYING`: The creation/deletion attempt has failed. It will be retried three times.
`FAILED`: The creation/deletion attempt failed after three retries. End the current execution period and go to the next execution period.
`TIME_LIMIT_EXCEEDED`: The creation/deletion time exceeded the time_limit set in the policy. End the current execution period and go to the next execution period. +`latest_execution.start_time` | The start time of the latest execution in milliseconds since the epoch. +`latest_execution.end_time` | The end time of the latest execution in milliseconds since the epoch. +`latest_execution.info.message` | A user-friendly message describing the status of the latest execution. +`latest_execution.info.cause` | Contains the failure reason if the latest execution fails. +`retry.count` | The number of remaining execution retry attempts. + + +## Start a policy +Introduced 2.1 +{: .label .label-purple } + +Starts the policy by setting its `enabled` flag to `true`. + +#### Request + +```json +POST _plugins/_sm/policies//_start +``` + +### Example + +```json +POST _plugins/_sm/policies/daily-policy/_start +``` + +### Response + +```json +{ + "acknowledged" : true +} +``` + +## Stop a policy +Introduced 2.1 +{: .label .label-purple } + +Sets the `enabled` flag to `false` for an SM policy. The policy will not run until you [start](#start-a-policy) it. + +#### Request + +```json +POST _plugins/_sm/policies//_stop +``` + +### Example + +```json +POST _plugins/_sm/policies/daily-policy/_stop +``` + +### Response + +```json +{ + "acknowledged" : true +} +``` + +## Delete a policy +Introduced 2.1 +{: .label .label-purple } + +Deletes the specified SM policy. + +#### Request + +```json +DELETE _plugins/_sm/policies/ +``` + +### Example + +```json +DELETE _plugins/_sm/policies/daily-policy +``` + +### Response + +```json +{ + "_index" : ".opendistro-ism-config", + "_id" : "daily-policy-sm-policy", + "_version" : 8, + "result" : "deleted", + "forced_refresh" : true, + "_shards" : { + "total" : 2, + "successful" : 2, + "failed" : 0 + }, + "_seq_no" : 45366, + "_primary_term" : 20 +} +``` \ No newline at end of file diff --git a/_opensearch/snapshots/sm-dashboards.md b/_opensearch/snapshots/sm-dashboards.md new file mode 100644 index 00000000..5844e708 --- /dev/null +++ b/_opensearch/snapshots/sm-dashboards.md @@ -0,0 +1,97 @@ +--- +layout: default +title: Snapshot Management in OpenSearch Dashboards +parent: Snapshots +nav_order: 35 +has_children: false +--- + +# Using Snapshot Management in OpenSearch Dashboards + +You can set up Snapshot Management (SM) in OpenSearch Dashboards. + +## Create a repository + +Before you create an SM policy, you need to set up a repository for snapshots. + +1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. +1. In the left panel, under **Snapshot Management**, select **Repositories**. +1. Select the **Create Repository** button. +1. Enter the repository name, type, and location. +1. (Optional) Select **Advanced Settings** and enter additional settings for this repository as a JSON object. Example: +```json +{ + "chunk_size": null, + "compress": false, + "max_restore_bytes_per_sec": "40m", + "max_snapshot_bytes_per_sec": "40m", + "readonly": false +} +``` +1. Select the **Add** button. + +## Create an SM policy + +Create an SM policy to set up automatic snapshots. An SM policy defines an automated snapshot creation schedule and an optional automated deletion schedule. + +1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. +1. In the left panel, under **Snapshot Management**, select **Snapshot Policies**. +1. Select the **Create Policy** button. +1. In the **Policy settings** section: + 1. Enter the policy name. + 1. (Optional) Enter the policy description. +1. In the **Source and destination** section: + 1. Select or enter source indexes either as a list or as an index pattern. + 1. Select a repository for snapshots. To [create a new repository](#create-a-repository), select the **Create** button. +1. In the **Snapshot schedule** section: + 1. Select the desired snapshot frequency or enter a custom cron expression for snapshot frequency. + 1. Select the start time and time zone. +1. In the **Retention period** section: + 1. Choose to retain all snapshots or specify retention conditions (the maximum age of retained snapshots). + 1. (Optional) In **Additional settings**, select the minimum and maximum number of retained snapshots, deletion frequency, and deletion start time. +1. In the **Notifications** section, select the snapshot activities you want to be notified about. +1. (Optional) In the **Advanced settings** section, select the desired options: + - **Include cluster state in snapshots** + - **Ignore unavailable indices** + - **Allow partial snapshots** +1. Select the **Create** button. + +## View, edit, or delete an SM policy + +You can view, edit, or delete an SM policy on the policy details page. + +1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. +1. In the left panel, under **Snapshot Management**, select **Snapshot Policies**. +1. Click on the **Policy name** of the policy you want to view, edit, or delete.
+The policy settings, snapshot schedule, snapshot retention period, notifications, and last creation and deletion are displayed in the policy details page.
If a snapshot creation or deletion fails, you can view information about the failure in the **Last Creation/Deletion** section. To view the failure message, click on the **cause** in the **Info** column. +1. To edit or delete the SM policy, select the **Edit** or **Delete** button. + +## Enable, disable, or delete SM policies + +1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. +1. In the left panel, under **Snapshot Management**, select **Snapshot Policies**. +1. Select one or more policies in the list. +1. To enable or disable selected SM policies, select the **Enable** or **Disable** button. To delete selected SM policies, in the **Actions** list, select the **Delete** option. + +## View snapshots + +1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. +1. In the left panel, under **Snapshot Management**, select **Snapshots**. +All automatically or manually taken snapshots appear in the list. +1. To view a snapshot, click on its **Name**. + +## Take a snapshot + +Use the steps below to take a snapshot manually. If you need to restore a snapshot, use the [restore snapshot API operation]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#restore-snapshots). + +1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. +1. In the left panel, under **Snapshot Management**, select **Snapshots**. +1. Select the **Take snapshot** button. +1. Enter the snapshot name. +1. Select or enter source indexes either as a list or as an index pattern. +1. Select a repository for the snapshot. +1. (Optional) In the **Advanced options** section, select the desired options: + - **Include cluster state in snapshots** + - **Ignore unavailable indices** + - **Allow partial snapshots** +1. Select the **Add** button. diff --git a/_opensearch/snapshots/snapshot-management.md b/_opensearch/snapshots/snapshot-management.md new file mode 100644 index 00000000..50f831c6 --- /dev/null +++ b/_opensearch/snapshots/snapshot-management.md @@ -0,0 +1,78 @@ +--- +layout: default +title: Snapshot Management +parent: Snapshots +nav_order: 20 +has_children: false +--- + +# Snapshot Management + +Snapshot Management (SM) lets you automate [taking snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#take-snapshots). To use this feature, you need to install the [Index Management (IM) Plugin]({{site.url}}{{site.baseurl}}/im-plugin). Snapshots store only incremental changes since the last snapshot. Thus, while taking an initial snapshot may be a heavy operation, subsequent snapshots have minimal overhead. To set up automatic snapshots, you have to create an SM policy with a desired SM schedule and configuration. + +When you create an SM policy, its document ID is given the name `-sm-policy`. Because of this, SM policies have to obey the following rules: + +- SM policies must have unique names. + +- You cannot update the policy name after its creation. + +SM-created snapshots have names in the format `--`. Two snapshots created by different policies at the same time always have different names because of the `` prefix. To avoid name collisions within the same policy, each snapshot's name contains a random string suffix. + +Each policy has associated metadata that stores the policy status. Snapshot Management saves SM policies and metadata in the system index and reads them from the system index. Thus, Snapshot Management depends on the OpenSearch cluster's indexing and searching functions. The policy's metadata keeps information about the latest creation and deletion only. The metadata is read before running every scheduled job so that SM can continue execution from the previous job's state. You can view the metadata using the [explain API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#explain). + +An SM schedule is a custom [cron]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron) expression. It consists of two parts: a creation schedule and a deletion schedule. You must set up a creation schedule that specifies the frequency and timing of snapshot creation. Optionally, you can set up a separate schedule for deleting snapshots. + +An SM configuration includes the indexes and repository for the snapshots and supports all parameters you can define when [creating a snapshot]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#take-snapshots) using the API. Additionally, you can specify the format and time zone for the date used in the snapshot's name. + + +## Performance + +One snapshot can contain as many indexes as there are in the cluster. We expect at most dozens of SM policies in one cluster, but a snapshot repository can safely scale to thousands of snapshots. However, to manage its metadata, a large repository requires more memory on the cluster manager node. + +Snapshot Management depends on the Job Scheduler plugin to schedule a job that is run periodically. Each SM policy corresponds to one SM-scheduled job. The scheduled job is lightweight, so the burden of SM depends on the snapshot creation frequency and the burden of running the snapshot operation itself. + +## Concurrency + +An SM policy does not support concurrent snapshot operations, since too many such operations may degrade the cluster. Snapshot operations (creation or deletion) are performed asynchronously. SM does not start a new operation until the previous asynchronous operation finishes. + +We don't recommend creating several SM policies with the same schedule and overlapping indexes in one cluster because it leads to concurrent snapshot creation on the same indexes and hinders performance. +{: .warning } + + +We don't recommend setting up the same repository for multiple SM policies with same schedule in different clusters, since it may cause a sudden spike of burden in this repository. +{: .warning } + +## Failure management + +If a snapshot operation fails, it is retried a maximum of three times. The failure message is saved in `metadata.latest_execution` and is overwritten when a subsequent snapshot operation starts. You can view the failure message using the [explain API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#explain). When using OpenSearch Dashboards, you can view the failure message on the [policy details page]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-dashboards#view-edit-or-delete-an-sm-policy). Possible reasons for failure include red index status and shard reallocation. + +## Security + +The Security plugin has two built-in roles for Snapshot Management actions: `snapshot_management_full_access` and `snapshot_management_read_access`. For descriptions of each, see [Predefined roles]({{site.url}}{{site.baseurl}}/security-plugin/access-control/users-roles#predefined-roles). + +The following table lists the required permissions for each Snapshot Management API. + +Function | API | Permission +:--- | :--- | :--- +Get policy | GET _plugins/_sm/policies
GET _plugins/_sm/policies/`policy_name` | cluster:admin/opensearch/snapshot_management/policy/get
cluster:admin/opensearch/snapshot_management/policy/search +Create/update policy | POST _plugins/_sm/policies/`policy_name`
PUT _plugins/_sm/policies/`policy_name`?if_seq_no=1&if_primary_term=1 | cluster:admin/opensearch/snapshot_management/policy/write +Delete policy | DELETE _plugins/_sm/policies/`policy_name` | cluster:admin/opensearch/snapshot_management/policy/delete +Explain | GET _plugins/_sm/policies/`policy_names`/_explain | cluster:admin/opensearch/snapshot_management/policy/explain +Start | POST _plugins/_sm/policies/`policy_name`/_start | cluster:admin/opensearch/snapshot_management/policy/start +Stop| POST _plugins/_sm/policies/`policy_name`/_stop | cluster:admin/opensearch/snapshot_management/policy/stop + + +## API + +The following table lists all [Snapshot Management API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api) functions. + +Function | API | Description +:--- | :--- | :--- +[Create policy]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#create-or-update-a-policy) | POST _plugins/_sm/policies/`policy_name` | Creates an SM policy. +[Update policy]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#create-or-update-a-policy) | PUT _plugins/_sm/policies/`policy_name`?if_seq_no=`sequence_number`&if_primary_term=`primary_term` | Modifies the `policy_name` policy. +[Get all policies]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#get-policies) | GET _plugins/_sm/policies | Returns all SM policies. +[Get the policy `policy_name`]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#get-policies) | GET _plugins/_sm/policies/`policy_name` | Returns the `policy_name` SM policy. +[Delete policy]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#delete-a-policy) | DELETE _plugins/_sm/policies/`policy_name` | Deletes the `policy_name` policy. +[Explain]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#explain) | GET _plugins/_sm/policies/`policy_names`/_explain | Provides the enabled/disabled status and the metadata for all policies specified by `policy_names`. +[Start policy]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#start-a-policy) | POST _plugins/_sm/policies/`policy_name`/_start | Starts the `policy_name` policy. +[Stop policy]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#stop-a-policy)| POST _plugins/_sm/policies/`policy_name`/_stop | Stops the `policy_name` policy. \ No newline at end of file diff --git a/_opensearch/snapshot-restore.md b/_opensearch/snapshots/snapshot-restore.md similarity index 95% rename from _opensearch/snapshot-restore.md rename to _opensearch/snapshots/snapshot-restore.md index 2aaf94f9..349f0d3d 100644 --- a/_opensearch/snapshot-restore.md +++ b/_opensearch/snapshots/snapshot-restore.md @@ -1,23 +1,21 @@ --- layout: default title: Take and restore snapshots -nav_order: 65 +parent: Snapshots +nav_order: 10 +has_children: false --- # Take and restore snapshots -Snapshots are backups of a cluster's indices and state. State includes cluster settings, node information, index metadata (mappings, settings, templates, etc.), and shard allocation. +Snapshots aren't instantaneous. They take time to complete and do not represent perfect point-in-time views of the cluster. While a snapshot is in progress, you can still index documents and send other requests to the cluster, but new documents and updates to existing documents generally aren't included in the snapshot. The snapshot includes primary shards as they existed when OpenSearch initiated the snapshot. Depending on the size of your snapshot thread pool, different shards might be included in the snapshot at slightly different times. -Snapshots have two main uses: +OpenSearch snapshots are incremental, meaning that they only store data that has changed since the last successful snapshot. The difference in disk usage between frequent and infrequent snapshots is often minimal. -- **Recovering from failure** - - For example, if cluster health goes red, you might restore the red indices from a snapshot. - -- **Migrating from one cluster to another** - - For example, if you're moving from a proof-of-concept to a production cluster, you might take a snapshot of the former and restore it on the latter. +In other words, taking hourly snapshots for a week (for a total of 168 snapshots) might not use much more disk space than taking a single snapshot at the end of the week. Also, the more frequently you take snapshots, the less time they take to complete. Some OpenSearch users take snapshots as often as every 30 minutes. +If you need to delete a snapshot, be sure to use the OpenSearch API rather than navigating to the storage location and purging files. Incremental snapshots from a cluster often share a lot of the same data; when you use the API, OpenSearch only removes data that no other snapshot is using. +{: .tip } --- @@ -28,18 +26,6 @@ Snapshots have two main uses: --- -## About snapshots - -Snapshots aren't instantaneous. They take time to complete and do not represent perfect point-in-time views of the cluster. While a snapshot is in progress, you can still index documents and make other requests to the cluster, but new documents and updates to existing documents generally aren't included in the snapshot. The snapshot includes primary shards as they existed when OpenSearch initiated the snapshot. Depending on the size of your snapshot thread pool, different shards might be included in the snapshot at slightly different times. - -OpenSearch snapshots are incremental, meaning that they only store data that has changed since the last successful snapshot. The difference in disk usage between frequent and infrequent snapshots is often minimal. - -In other words, taking hourly snapshots for a week (for a total of 168 snapshots) might not use much more disk space than taking a single snapshot at the end of the week. Also, the more frequently you take snapshots, the less time they take to complete. Some OpenSearch users take snapshots as often as every half hour. - -If you need to delete a snapshot, be sure to use the OpenSearch API rather than navigating to the storage location and purging files. Incremental snapshots from a cluster often share a lot of the same data; when you use the API, OpenSearch only removes data that no other snapshot is using. -{: .tip } - - ## Register repository Before you can take a snapshot, you have to "register" a snapshot repository. A snapshot repository is just a storage location: a shared file system, Amazon S3, Hadoop Distributed File System (HDFS), Azure Storage, etc. diff --git a/_security-plugin/access-control/multi-tenancy.md b/_security-plugin/access-control/multi-tenancy.md index 46c7cd40..d1b8ab04 100644 --- a/_security-plugin/access-control/multi-tenancy.md +++ b/_security-plugin/access-control/multi-tenancy.md @@ -165,4 +165,4 @@ The open source version of OpenSearch Dashboards saves all objects to a single i The security plugin scrubs these index names of special characters, so they might not be a perfect match of tenant names and usernames. {: .tip } -To back up your OpenSearch Dashboards data, [take a snapshot]({{site.url}}{{site.baseurl}}/opensearch/snapshot-restore/) of all tenant indices using an index pattern such as `.kibana*`. +To back up your OpenSearch Dashboards data, [take a snapshot]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore/) of all tenant indexes using an index pattern such as `.kibana*`. diff --git a/_security-plugin/access-control/users-roles.md b/_security-plugin/access-control/users-roles.md index 9b8e45eb..cc5abd87 100644 --- a/_security-plugin/access-control/users-roles.md +++ b/_security-plugin/access-control/users-roles.md @@ -124,9 +124,10 @@ Role | Description `reports_instances_read_access` | Grants permissions to generate on-demand reports and download existing reports, but not to view or create report definitions. `reports_full_access` | Grants full permissions to reports. `asynchronous_search_full_access` | Grants full permissions to all asynchronous search actions. -`asynchronous_search_read_access` | Grants permissions to view asynchronous searches, but not to submit, modify, or delete async searches. +`asynchronous_search_read_access` | Grants permissions to view asynchronous searches but not to submit, modify, or delete async searches. `index_management_full_access` | Grants full permissions to all index management actions, including ISM, transforms, and rollups. - +`snapshot_management_full_access` | Grants full permissions to all snapshot management actions. +`snapshot_management_read_access` | Grants permissions to view policies but not create, modify, start, stop, or delete them. For more detailed summaries of the permissions for each role, reference their action groups against the descriptions in [Default action groups]({{site.url}}{{site.baseurl}}/security-plugin/access-control/default-action-groups/). diff --git a/_upgrade-to/index.md b/_upgrade-to/index.md index 01c7fc3a..763c21cf 100644 --- a/_upgrade-to/index.md +++ b/_upgrade-to/index.md @@ -16,6 +16,6 @@ Three approaches exist: - Perform a [rolling upgrade or cluster restart upgrade]({{site.url}}{{site.baseurl}}/upgrade-to/upgrade-to/) on your existing nodes. - Replace existing Elasticsearch OSS nodes with new OpenSearch nodes. Node replacement is most popular when upgrading [Docker clusters]({{site.url}}{{site.baseurl}}/upgrade-to/docker-upgrade-to/). -Regardless of your approach, to safeguard against data loss, we recommend that you take a [snapshot]({{site.url}}{{site.baseurl}}/opensearch/snapshot-restore/) of all indices prior to any migration. +Regardless of your approach, to safeguard against data loss, we recommend that you take a [snapshot]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore) of all indexes prior to any migration. If your existing clients include a version check, such as recent versions of Logstash OSS and Filebeat OSS, [check compatibility]({{site.url}}{{site.baseurl}}/clients/agents-and-ingestion-tools/index/) before upgrading. diff --git a/_upgrade-to/snapshot-migrate.md b/_upgrade-to/snapshot-migrate.md index e95fa681..7d7bbbf7 100644 --- a/_upgrade-to/snapshot-migrate.md +++ b/_upgrade-to/snapshot-migrate.md @@ -6,6 +6,6 @@ nav_order: 5 # Use snapshots to migrate data -One popular approach is to take a [snapshot]({{site.url}}{{site.baseurl}}/opensearch/snapshot-restore/) of your Elasticsearch OSS 6.x or 7.x indices, [create an OpenSearch cluster]({{site.url}}{{site.baseurl}}/opensearch/install/), restore the snapshot on the new cluster, and point your clients to the new host. +One popular approach is to take a [snapshot]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore) of your Elasticsearch OSS 6.x or 7.x indexes, [create an OpenSearch cluster]({{site.url}}{{site.baseurl}}/opensearch/install/), restore the snapshot on the new cluster, and point your clients to the new host. The snapshot approach can mean running two clusters in parallel, but lets you validate that the OpenSearch cluster is working in a way that meets your needs prior to modifying the Elasticsearch OSS cluster. diff --git a/images/SM-state-machine.PNG b/images/SM-state-machine.PNG new file mode 100644 index 00000000..312edd0d Binary files /dev/null and b/images/SM-state-machine.PNG differ