diff --git a/docs/reference/index-modules.asciidoc b/docs/reference/index-modules.asciidoc index a2f7c242170..3531aaa5df0 100644 --- a/docs/reference/index-modules.asciidoc +++ b/docs/reference/index-modules.asciidoc @@ -95,7 +95,7 @@ indices. Below is a list of all _dynamic_ index settings that are not associated with any specific index module: - +[[dynamic-index-number-of-replicas]] `index.number_of_replicas`:: The number of replicas each primary shard has. Defaults to 1. diff --git a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc new file mode 100644 index 00000000000..a333b45741b --- /dev/null +++ b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc @@ -0,0 +1,210 @@ +[[restore-snapshot-api]] +=== Restore snapshot API +++++ +Restore snapshot +++++ + +Restores a <> of a cluster or specified data streams and indices. + +//// +[source,console] +---- +PUT /_snapshot/my_repository +{ + "type": "fs", + "settings": { + "location": "my_backup_location" + } +} + +PUT /_snapshot/my_repository/my_snapshot?wait_for_completion=true + +PUT /index_1 + +PUT /index_2 + +PUT /index_3 + +PUT /index_4 + +PUT _snapshot/my_repository/snapshot_2?wait_for_completion=true +{ + "indices": "index_3,index_4", + "ignore_unavailable": true, + "include_global_state": false, + "metadata": { + "taken_by": "Elastic Machine", + "taken_because": "backup testing" + } +} + +POST /index_1/_close + +POST /index_2/_close + +POST /index_3/_close + +POST /index_4/_close + +---- +// TESTSETUP +//// + +[source,console] +---- +POST /_snapshot/my_repository/my_snapshot/_restore +---- +// TEST[s/_restore/_restore?wait_for_completion=true/] + +[[restore-snapshot-api-request]] +==== {api-request-title} + +`POST /_snapshot//` + +[[restore-snapshot-api-desc]] +==== {api-description-title} + +Use the restore snapshot API to restore a snapshot of a cluster, including all data streams and indices in the snapshot. If you do not want to restore the entire snapshot, you can select specific data streams or indices to restore. + +NOTE: You cannot restore a data stream if a stream with the same name already +exists. + +You can run the restore operation on a cluster that contains an elected +<> and has data nodes with enough capacity to accommodate the snapshot +you are restoring. Existing indices can only be restored if they are +<> and have the same number of shards as the indices in +the snapshot. The restore operation automatically opens restored indices if +they were closed and creates new indices if they do not exist in the cluster. + +If a data stream is restored, its backing indices are also restored. +Alternatively, you can restore individual backing indices without restoring an +entire data stream. If you restore individual backing indices, they are not +automatically added to any existing data stream. For example, if only the +`.ds-logs-00003` backing index is restored from a snapshot, it is not +automatically added to the existing `logs` data stream. + +include::{es-ref-dir}/snapshot-restore/restore-snapshot.asciidoc[tag=index-settings-data-stream-warning] + +[[restore-snapshot-api-path-params]] +==== {api-path-parms-title} + +``:: +(Required, string) +Name of the repository to restore a snapshot from. + +``:: +(Required, string) +Name of the snapshot to restore. + +[role="child-attributes"] +[[restore-snapshot-api-request-body]] +==== {api-request-body-title} + +`ignore_index_settings`:: +(Optional, string) +A comma-separated list of index settings that should not be restored from a snapshot. + +`include_aliases`:: +(Optional, boolean) +If `true`, index aliases from the original snapshot are restored. +Defaults to `true`. ++ +If `false`, prevents aliases from being restored together with associated +indices. + +[[restore-snapshot-api-include-global-state]] +`include_global_state`:: ++ +-- +(Optional, boolean) +If `false`, the cluster state is not restored. Defaults to `false`. + +If `true`, the current cluster state is included in the restore operation. + +The cluster state includes: + +* Persistent cluster settings +* Index templates +* Legacy index templates +* Ingest pipelines +* {ilm-init} lifecycle policies +-- ++ +IMPORTANT: By default, the entire restore operation will fail if one or more indices included in the snapshot do not have all primary shards available. You can change this behavior by setting <> to `true`. + +[[restore-snapshot-api-index-settings]] +`index_settings`:: +(Optional, string) +A comma-separated list of settings to add or change in all restored indices. Use this parameter to override index settings when restoring snapshots. ++ +For data streams, these index settings are applied to the restored backing +indices. ++ +For more information regarding all the different index-level settings +that you can specify, see +<>. + +`indices`:: +(Optional, string) +A comma-separated list of data streams and indices to restore from the snapshot. +<> is supported. ++ +By default, a restore operation includes all data streams and indices in the snapshot. If this +argument is provided, the restore operation only includes the specified data streams and indices. + +[[restore-snapshot-api-partial]] +`partial`:: +(Optional, boolean) +If `false`, the entire restore operation will fail if one or more indices included in the snapshot do not have all primary shards available. Defaults to `false`. ++ +If `true`, allows restoring a partial snapshot of indices with unavailable shards. Only shards that were successfully included in the snapshot +will be restored. All missing shards will be recreated as empty. + +[[restore-snapshot-api-rename-pattern]] +`rename_pattern`:: +(Optional, string) +Defines a rename pattern to apply to restored data streams and indices. Data streams and indices matching the rename pattern will be renamed according to +<>. ++ +The rename pattern is applied as defined by the regular expression that +supports referencing the original text, according to the https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#appendReplacement-java.lang.StringBuffer-java.lang.String-[`appendReplacement`] logic. ++ +The request will fail if two or more data streams or indices will be renamed into the same name. ++ +include::{es-ref-dir}/snapshot-restore/restore-snapshot.asciidoc[tag=rename-restored-data-stream-tag] + +[[restore-snapshot-api-rename-replacement]] +`rename_replacement`:: +(Optional, string) +Defines the rename replacement string. See <> for more information. + +`wait_for_completion`:: +(Optional, boolean) +If `false`, the request returns a response when the restore operation initializes. +Defaults to `false`. ++ +If `true`, the request returns a response when the restore operation +completes. + +[[restore-snapshot-api-example]] +==== {api-examples-title} + +The following request restores `index_1` and `index_2` from `snapshot_2`. The `rename_pattern` and `rename_replacement` parameters indicate any index matching the regular expression `index_(.+)` will be renamed using the pattern `restored_index_$1` when restored. + +For example, `index_1` will be renamed to `restored_index_1`. `index_2` will be renamed to `restored_index_2`. + +[source,console] +---- +POST /_snapshot/my_repository/snapshot_2/_restore?wait_for_completion=true +{ + "indices": "index_1,index_2", + "ignore_unavailable": true, + "include_global_state": false, + "rename_pattern": "index_(.+)", + "rename_replacement": "restored_index_$1", + "include_aliases": false +} +---- + +The API returns an acknowledgement if the request succeeds. If the request encounters errors, the response indicates any issues found, such as +open indices that are blocking the restore operation from completing. diff --git a/docs/reference/snapshot-restore/apis/snapshot-restore-apis.asciidoc b/docs/reference/snapshot-restore/apis/snapshot-restore-apis.asciidoc index fe91601a71b..8848348316e 100644 --- a/docs/reference/snapshot-restore/apis/snapshot-restore-apis.asciidoc +++ b/docs/reference/snapshot-restore/apis/snapshot-restore-apis.asciidoc @@ -6,12 +6,6 @@ backups, and restore snapshots to a running cluster. For more information, see <>. -[NOTE] -==== -We are working on including more snapshot and restore APIs in this section. Some -content may not be included yet. -==== - [discrete] [[snapshot-restore-repo-apis]] === Snapshot repository management APIs @@ -27,6 +21,7 @@ content may not be included yet. * <> * <> * <> +* <> * <> include::put-repo-api.asciidoc[] @@ -37,4 +32,5 @@ include::clean-up-repo-api.asciidoc[] include::create-snapshot-api.asciidoc[] include::get-snapshot-api.asciidoc[] include::get-snapshot-status-api.asciidoc[] +include::restore-snapshot-api.asciidoc[] include::delete-snapshot-api.asciidoc[] diff --git a/docs/reference/snapshot-restore/index.asciidoc b/docs/reference/snapshot-restore/index.asciidoc index 37190dfc405..daeb2f8f5ed 100644 --- a/docs/reference/snapshot-restore/index.asciidoc +++ b/docs/reference/snapshot-restore/index.asciidoc @@ -10,10 +10,14 @@ You can take snapshots of an entire cluster, including all its data streams and indices. You can also take snapshots of only specific data streams or indices in the cluster. +You must +<> +before you can <>. + Snapshots can be stored in either local or remote repositories. Remote repositories can reside on Amazon S3, HDFS, Microsoft Azure, Google Cloud Storage, -and other platforms supported by a repository plugin. +and other platforms supported by a {plugins}/repository.html[repository plugin]. Snapshots are incremental: each snapshot only stores data that is not part of an earlier snapshot. @@ -21,16 +25,14 @@ This enables you to take frequent snapshots with minimal overhead. // end::snapshot-intro[] // tag::restore-intro[] -You can restore snapshots to a running cluster with the <>. -By default, all data streams and indices in the snapshot are restored. +You can <> to a running cluster, which includes all data streams and indices in the snapshot +by default. However, you can choose to restore only the cluster state or specific data streams or indices from a snapshot. // end::restore-intro[] -You must <> -before you can <>. - -You can use <> +You can use +<> to automatically take and manage snapshots. // tag::backup-warning[] diff --git a/docs/reference/snapshot-restore/register-repository.asciidoc b/docs/reference/snapshot-restore/register-repository.asciidoc index b0140da070e..03e0b344511 100644 --- a/docs/reference/snapshot-restore/register-repository.asciidoc +++ b/docs/reference/snapshot-restore/register-repository.asciidoc @@ -7,19 +7,23 @@ [[snapshots-register-repository-description]] // tag::snapshots-register-repository-tag[] You must register a snapshot repository before you can perform snapshot and -restore operations. We recommend creating a new snapshot repository for each +restore operations. Use the <> to register or update a snapshot repository. We recommend creating a new snapshot repository for each major version. The valid repository settings depend on the repository type. -If you register same snapshot repository with multiple clusters, only +If you register the same snapshot repository with multiple clusters, only one cluster should have write access to the repository. All other clusters connected to that repository should set the repository to `readonly` mode. // end::snapshots-register-repository-tag[] -IMPORTANT: The snapshot format can change across major versions, so if you have -clusters on different versions trying to write the same repository, snapshots + +[IMPORTANT] +==== +The snapshot format can change across major versions, so if you have +clusters on different versions trying to write to the same repository, snapshots written by one version may not be visible to the other and the repository could be corrupted. While setting the repository to `readonly` on all but one of the clusters should work with multiple clusters differing by one major version, it is not a supported configuration. +==== [source,console] ----------------------------------- @@ -33,14 +37,14 @@ PUT /_snapshot/my_backup ----------------------------------- // TESTSETUP -To retrieve information about a registered repository, use a GET request: +Use the <> to retrieve information about a registered repository: [source,console] ----------------------------------- GET /_snapshot/my_backup ----------------------------------- -which returns: +This request returns the following response: [source,console-result] ----------------------------------- @@ -55,7 +59,7 @@ which returns: ----------------------------------- To retrieve information about multiple repositories, specify a comma-delimited -list of repositories. You can also use the * wildcard when +list of repositories. You can also use a wildcard (`*`) when specifying repository names. For example, the following request retrieves information about all of the snapshot repositories that start with `repo` or contain `backup`: @@ -66,22 +70,22 @@ GET /_snapshot/repo*,*backup* ----------------------------------- To retrieve information about all registered snapshot repositories, omit the -repository name or specify `_all`: +repository name: [source,console] ----------------------------------- GET /_snapshot ----------------------------------- -or +Alternatively, you can specify `_all`: [source,console] ----------------------------------- GET /_snapshot/_all ----------------------------------- -You can unregister a repository using the <>: +You can unregister a repository using the +<>: [source,console] ----------------------------------- @@ -151,7 +155,6 @@ PUT /_snapshot/my_fs_backup The following settings are supported: -[horizontal] `location`:: Location of the snapshots. Mandatory. `compress`:: Turns on compression of the snapshot files. Compression is applied only to metadata files (index mapping and settings). Data files are not compressed. Defaults to `true`. `chunk_size`:: Big files can be broken down into chunks during snapshotting if needed. Specify the chunk size as a value and diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 3e2cb8f43ce..a3c224b4634 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -18,18 +18,19 @@ PUT /_snapshot/my_backup/snapshot_1?wait_for_completion=true //// -A snapshot can be restored using the following command: +Use the <> to restore +a snapshot of a cluster or specified data streams and indices: [source,console] ----------------------------------- POST /_snapshot/my_backup/snapshot_1/_restore ----------------------------------- +// TEST[s/_restore/_restore?wait_for_completion=true/] By default, all data streams and indices in the snapshot are restored, but the cluster state is -*not* restored. It's possible to select specific data streams or indices that should be restored as well -as to allow the global cluster state from being restored by using `indices` and -`include_global_state` options in the restore request body. The list -supports <>. +*not* restored. Use the `indices` parameter to restore only specific data streams or indices. This parameter +supports <>. To include the global cluster state, set +`include_global_state` to `true` in the restore request body. [WARNING] ==== @@ -52,10 +53,10 @@ If no index template matches a data stream, the stream cannot The `rename_pattern` and `rename_replacement` options can be also used to rename data streams and indices on restore -using regular expression that supports referencing the original text as -explained -http://docs.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html#appendReplacement(java.lang.StringBuffer,%20java.lang.String)[here]. +using regular expression that supports referencing the original text, according to the https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#appendReplacement-java.lang.StringBuffer-java.lang.String-[`appendReplacement`] logic. +[[rename-restored-data-stream]] +// tag::rename-restored-data-stream-tag[] If you rename a restored data stream, its backing indices are also renamed. For example, if you rename the `logs` data stream to `restored-logs`, the backing index `.ds-logs-000005` is renamed to `.ds-restored-logs-000005`. @@ -66,6 +67,7 @@ If you rename a restored stream, ensure an index template matches the new stream name. If no index template matches the stream, it cannot <> or create new backing indices. ==== +// end::rename-restored-data-stream-tag[] Set `include_aliases` to `false` to prevent aliases from being restored together with associated indices @@ -92,8 +94,8 @@ pipelines, and {ilm-init} policies are restored into the current cluster. This overwrites any existing cluster settings, templates, pipelines and {ilm-init} policies whose names match those in the snapshot. -The restore operation can be performed on a functioning cluster. However, an -existing index can be only restored if it's <> and +The restore operation must be performed on a functioning cluster. However, an +existing index can be only restored if it's <> and has the same number of shards as the index in the snapshot. The restore operation automatically opens restored indices if they were closed and creates new indices if they didn't exist in the cluster. @@ -110,7 +112,7 @@ added to any existing data streams. For example, if only the `.ds-logs-000003` backing index is restored from a snapshot, it is not automatically added to the existing `logs` data stream. -[float] +[discrete] === Partial restore By default, the entire restore operation will fail if one or more indices or backing indices participating in the operation don't have @@ -119,11 +121,13 @@ restore such indices by setting `partial` to `true`. Please note, that only succ restored in this case and all missing shards will be recreated empty. -[float] +[discrete] === Changing index settings during restore -Most of index settings can be overridden during the restore process. For example, the following command will restore -the index `index_1` without creating any replicas while switching back to default refresh interval: +Use the <> parameter +to override index settings during the restore process. For example, the +following request will restore the index `index_1` without creating any +replicas while switching back to the default refresh interval: [source,console] ----------------------------------- @@ -141,11 +145,12 @@ POST /_snapshot/my_backup/snapshot_1/_restore ----------------------------------- // TEST[continued] -Please note, that some settings such as `index.number_of_shards` cannot be changed during restore operation. +NOTE: Some settings such as `index.number_of_shards` cannot be changed during the restore operation. For data streams, these index settings are applied to the restored backing indices. +// tag::index-settings-data-stream-warning[] [IMPORTANT] ==== The `index_settings` and `ignore_index_settings` parameters affect @@ -157,25 +162,27 @@ If you change index settings during a restore, we recommend you make similar changes in the stream's matching index template. This ensures new backing indices created for the stream use the same index settings. ==== +// end::index-settings-data-stream-warning[] -[float] +[discrete] === Restoring to a different cluster -The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore it's possible to -restore a snapshot made from one cluster into another cluster. All that is required is registering the repository -containing the snapshot in the new cluster and starting the restore process. The new cluster doesn't have to have the -same size or topology. However, the version of the new cluster should be the same or newer (only 1 major version newer) than the cluster that was used to create the snapshot. For example, you can restore a 1.x snapshot to a 2.x cluster, but not a 1.x snapshot to a 5.x cluster. +The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore, it's possible to +restore a snapshot created from one cluster into another cluster by registering the repository that contains the snapshot in the new cluster and starting the restore process. -If the new cluster has a smaller size additional considerations should be made. First of all it's necessary to make sure -that new cluster have enough capacity to store all data streams and indices in the snapshot. It's possible to change index settings -during restore to reduce the number of replicas, which can help with restoring snapshots into smaller cluster. It's also -possible to select only subset of the data streams or indices using the `indices` parameter. +The topology of the new cluster does not have to match the cluster where the snapshot was created. However, the version of the new cluster must be the same or only one major version newer than the cluster that was used to create the snapshot. For example, you can restore a 5.x snapshot to a 6.x cluster, but not a 5.x snapshot to a 7.x cluster. + +If the new cluster has a smaller capacity, consider the following changes: + +* Ensure that the new cluster has enough capacity to store all data streams and indices in the snapshot. +* Change the index settings during the restore operation to reduce the +<>. +* Use the `indices` parameter to choose only specific data streams or indices to restore. If indices or backing indices in the original cluster were assigned to particular nodes using -<>, the same rules will be enforced in the new cluster. Therefore -if the new cluster doesn't contain nodes with appropriate attributes that a restored index can be allocated on, such -index will not be successfully restored unless these index allocation settings are changed during restore operation. +<>, the same rules will be enforced in the new cluster. If the new cluster does not contain nodes with appropriate attributes that a restored index can be allocated on, the +index will not be successfully restored unless these index allocation settings are changed during the restore operation. The restore operation also checks that restored persistent settings are compatible with the current cluster to avoid accidentally restoring incompatible settings. If you need to restore a snapshot with incompatible persistent settings, try restoring it without -the global cluster state. +the <>.