2015-09-03 13:12:52 -04:00
[[repository-azure]]
=== Azure Repository Plugin
The Azure Repository plugin adds support for using Azure as a repository for
{ref}/modules-snapshots.html[Snapshot/Restore].
2017-04-20 09:01:37 -04:00
:plugin_name: repository-azure
include::install_remove.asciidoc[]
2015-09-03 13:12:52 -04:00
[[repository-azure-usage]]
==== Azure Repository
2017-02-28 06:25:01 -05:00
To enable Azure repositories, you have first to define your azure storage settings as
2018-08-01 05:07:23 -04:00
{ref}/secure-settings.html[secure settings], before starting up the node:
2015-09-03 13:12:52 -04:00
2017-02-28 06:25:01 -05:00
[source,sh]
----------------------------------------------------------------
bin/elasticsearch-keystore add azure.client.default.account
bin/elasticsearch-keystore add azure.client.default.key
----------------------------------------------------------------
2015-08-31 15:44:48 -04:00
Note that you can also define more than one account:
2015-09-03 13:12:52 -04:00
2017-02-28 06:25:01 -05:00
[source,sh]
----------------------------------------------------------------
bin/elasticsearch-keystore add azure.client.default.account
bin/elasticsearch-keystore add azure.client.default.key
bin/elasticsearch-keystore add azure.client.secondary.account
2019-06-26 07:43:32 -04:00
bin/elasticsearch-keystore add azure.client.secondary.sas_token
2017-02-28 06:25:01 -05:00
----------------------------------------------------------------
2015-09-03 13:12:52 -04:00
2019-09-05 13:44:21 -04:00
For more information about these settings, see
2020-07-14 16:31:07 -04:00
<<repository-azure-client-settings>>.
2015-11-28 06:59:09 -05:00
2017-06-12 20:03:18 -04:00
[IMPORTANT]
.Supported Azure Storage Account types
===============================================
The Azure Repository plugin works with all Standard storage accounts
* Standard Locally Redundant Storage - `Standard_LRS`
* Standard Zone-Redundant Storage - `Standard_ZRS`
* Standard Geo-Redundant Storage - `Standard_GRS`
* Standard Read Access Geo-Redundant Storage - `Standard_RAGRS`
https://azure.microsoft.com/en-gb/documentation/articles/storage-premium-storage[Premium Locally Redundant Storage] (`Premium_LRS`) is **not supported** as it is only usable as VM disk storage, not as general storage.
===============================================
2019-09-05 13:44:21 -04:00
[[repository-azure-client-settings]]
==== Client settings
2017-09-13 05:51:55 -04:00
2019-09-05 13:44:21 -04:00
The client that you use to connect to Azure has a number of settings available.
The settings have the form `azure.client.CLIENT_NAME.SETTING_NAME`. By default,
`azure` repositories use a client named `default`, but this can be modified using
the <<repository-azure-repository-settings,repository setting>> `client`.
For example:
2019-09-06 11:31:13 -04:00
[source,console]
2017-09-13 05:51:55 -04:00
----
2019-09-05 13:44:21 -04:00
PUT _snapshot/my_backup
{
"type": "azure",
"settings": {
"client": "secondary"
}
}
2017-09-13 05:51:55 -04:00
----
2019-09-05 13:44:21 -04:00
// TEST[skip:we don't have azure setup while testing this]
Most client settings can be added to the `elasticsearch.yml` configuration file.
For example:
2017-09-13 05:51:55 -04:00
2019-09-05 13:44:21 -04:00
[source,yaml]
----
azure.client.default.timeout: 10s
azure.client.default.max_retries: 7
azure.client.default.endpoint_suffix: core.chinacloudapi.cn
azure.client.secondary.timeout: 30s
----
2017-09-13 05:51:55 -04:00
2019-09-05 13:44:21 -04:00
In this example, the client side timeout is `10s` per try for the `default`
account with `7` retries before failing. The endpoint suffix is
`core.chinacloudapi.cn` and `30s` per try for the `secondary` account with `3`
retries.
The `account`, `key`, and `sas_token` storage settings are reloadable secure
settings, which you add to the {es} keystore. For more information about
creating and updating the {es} keystore, see
{ref}/secure-settings.html[Secure settings]. After you reload the settings, the
internal Azure clients, which are used to transfer the snapshot, utilize the
latest settings from the keystore.
NOTE: In progress snapshot or restore jobs will not be preempted by a *reload*
of the storage secure settings. They will complete using the client as it was
built when the operation started.
The following list contains the available client settings. Those that must be
stored in the keystore are marked as "secure"; the other settings belong in the
`elasticsearch.yml` file.
`account` ({ref}/secure-settings.html[Secure], {ref}/secure-settings.html#reloadable-secure-settings[reloadable])::
The Azure account name, which is used by the repository's internal Azure client.
2020-07-14 16:31:07 -04:00
2019-09-05 13:44:21 -04:00
`endpoint_suffix`::
The Azure endpoint suffix to connect to. The default value is
`core.windows.net`.
`key` ({ref}/secure-settings.html[Secure], {ref}/secure-settings.html#reloadable-secure-settings[reloadable])::
The Azure secret key, which is used by the repository's internal Azure client. Alternatively, use `sas_token`.
`max_retries`::
The number of retries to use when an Azure request fails. This setting helps
control the exponential backoff policy. It specifies the number of retries
that must occur before the snapshot fails. The default value is `3`. The
initial backoff period is defined by Azure SDK as `30s`. Thus there is `30s`
of wait time before retrying after a first timeout or failure. The maximum
backoff period is defined by Azure SDK as `90s`.
`proxy.host`::
The host name of a proxy to connect to Azure through. For example: `azure.client.default.proxy.host: proxy.host`.
`proxy.port`::
The port of a proxy to connect to Azure through. For example, `azure.client.default.proxy.port: 8888`.
`proxy.type`::
Register a proxy type for the client. Supported values are `direct`, `http`,
and `socks`. For example: `azure.client.default.proxy.type: http`. When
`proxy.type` is set to `http` or `socks`, `proxy.host` and `proxy.port` must
also be provided. The default value is `direct`.
`sas_token` ({ref}/secure-settings.html[Secure], {ref}/secure-settings.html#reloadable-secure-settings[reloadable])::
A shared access signatures (SAS) token, which the repository's internal Azure
client uses for authentication. The SAS token must have read (r), write (w),
list (l), and delete (d) permissions for the repository base path and all its
contents. These permissions must be granted for the blob service (b) and apply
to resource types service (s), container (c), and object (o). Alternatively,
use `key`.
`timeout`::
The client side timeout for any single request to Azure. The value should
specify the time unit. For example, a value of `5s` specifies a 5 second
timeout. There is no default value, which means that {es} uses the
http://azure.github.io/azure-storage-java/com/microsoft/azure/storage/RequestOptions.html#setTimeoutIntervalInMs(java.lang.Integer)[default value]
set by the Azure client (known as 5 minutes). This setting can be defined
globally, per account, or both.
2017-09-13 05:51:55 -04:00
2015-12-01 05:35:56 -05:00
[[repository-azure-repository-settings]]
2018-05-22 03:19:07 -04:00
==== Repository settings
2015-08-31 15:44:48 -04:00
2015-09-03 13:12:52 -04:00
The Azure repository supports following settings:
2017-02-28 06:25:01 -05:00
`client`::
2015-08-31 15:44:48 -04:00
2017-02-28 06:25:01 -05:00
Azure named client to use. Defaults to `default`.
2015-08-31 15:44:48 -04:00
2015-09-03 13:12:52 -04:00
`container`::
2017-01-30 05:47:08 -05:00
Container name. You must create the azure container before creating the repository.
Defaults to `elasticsearch-snapshots`.
2015-09-03 13:12:52 -04:00
`base_path`::
Specifies the path within container to repository data. Defaults to empty
(root directory).
`chunk_size`::
2020-07-14 16:31:07 -04:00
Big files can be broken down into multiple smaller blobs in the blob store during snapshotting.
It is not recommended to change this value from its default unless there is an explicit reason for limiting the
size of blobs in the repository. Setting a value lower than the default can result in an increased number of API
calls to the Azure blob store during snapshot create as well as restore operations compared to using the default
value and thus make both operations slower as well as more costly.
2019-08-14 16:59:36 -04:00
Specify the chunk size as a value and unit, for example:
2020-07-14 16:31:07 -04:00
`10MB`, `5KB`, `500B`. Defaults to the maximum size of a blob in the Azure blob store which is `5TB`.
2015-09-03 13:12:52 -04:00
`compress`::
When set to `true` metadata files are stored in compressed format. This
setting doesn't affect index files that are already compressed by default.
Defaults to `false`.
2019-03-28 15:03:14 -04:00
include::repository-shared-settings.asciidoc[]
2015-09-03 13:12:52 -04:00
2015-08-31 15:44:48 -04:00
`location_mode`::
`primary_only` or `secondary_only`. Defaults to `primary_only`. Note that if you set it
2016-12-08 10:57:50 -05:00
to `secondary_only`, it will force `readonly` to true.
2015-08-31 15:44:48 -04:00
2015-09-03 13:12:52 -04:00
Some examples, using scripts:
2019-09-09 13:38:14 -04:00
[source,console]
2015-09-03 13:12:52 -04:00
----
2019-01-07 08:44:12 -05:00
# The simplest one
2015-09-03 13:12:52 -04:00
PUT _snapshot/my_backup1
{
"type": "azure"
}
# With some settings
PUT _snapshot/my_backup2
{
"type": "azure",
"settings": {
2015-11-23 07:14:02 -05:00
"container": "backup-container",
2015-09-03 13:12:52 -04:00
"base_path": "backups",
2019-08-14 16:59:36 -04:00
"chunk_size": "32MB",
2015-09-03 13:12:52 -04:00
"compress": true
}
}
2015-08-31 15:44:48 -04:00
# With two accounts defined in elasticsearch.yml (my_account1 and my_account2)
PUT _snapshot/my_backup3
{
"type": "azure",
"settings": {
2017-02-28 06:25:01 -05:00
"client": "secondary"
2015-08-31 15:44:48 -04:00
}
}
PUT _snapshot/my_backup4
{
"type": "azure",
"settings": {
2017-02-28 06:25:01 -05:00
"client": "secondary",
2015-08-31 15:44:48 -04:00
"location_mode": "primary_only"
}
}
2015-09-03 13:12:52 -04:00
----
2016-05-13 16:15:51 -04:00
// TEST[skip:we don't have azure setup while testing this]
2015-09-03 13:12:52 -04:00
Example using Java:
[source,java]
----
2015-08-31 15:44:48 -04:00
client.admin().cluster().preparePutRepository("my_backup_java1")
2017-03-16 06:20:42 -04:00
.setType("azure").setSettings(Settings.builder()
2015-11-23 07:14:02 -05:00
.put(Storage.CONTAINER, "backup-container")
2015-09-03 13:12:52 -04:00
.put(Storage.CHUNK_SIZE, new ByteSizeValue(32, ByteSizeUnit.MB))
).get();
----
[[repository-azure-validation]]
2018-05-22 03:19:07 -04:00
==== Repository validation rules
2015-09-03 13:12:52 -04:00
According to the http://msdn.microsoft.com/en-us/library/dd135715.aspx[containers naming guide], a container name must
be a valid DNS name, conforming to the following naming rules:
* Container names must start with a letter or number, and can contain only letters, numbers, and the dash (-) character.
* Every dash (-) character must be immediately preceded and followed by a letter or number; consecutive dashes are not
permitted in container names.
* All letters in a container name must be lowercase.
* Container names must be from 3 through 63 characters long.