honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[DOC] Repository GCS ADC not supported (#33238) 

Make it clear that automatic default credentials (ADC)
is not supported for the repository-gcs plugin.
"Service Account" method is the only alternative
to authn requests to Google Cloud Storage.
This commit is contained in:
Albert Zaharovits 2018-08-30 10:32:08 +03:00 committed by GitHub
parent 9c541b8f72
commit c31c51dc80
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 70 additions and 69 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

139
docs/plugins/repository-gcs.asciidoc
View File

@ -10,71 +10,66 @@ include::install_remove.asciidoc[]
[[repository-gcs-usage]] [[repository-gcs-usage]]
==== Getting started ==== Getting started
The plugin uses the https://cloud.google.com/storage/docs/json_api/[Google Cloud Storage JSON API] (v1) The plugin uses the https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-clients/google-cloud-storage[Google Cloud Java Client for Storage]
to connect to the Storage service. If this is the first time you use Google Cloud Storage, you first to connect to the Storage service. If you are using
need to connect to the https://console.cloud.google.com/[Google Cloud Platform Console] and create a new https://cloud.google.com/storage/[Google Cloud Storage] for the first time, you
project. Once your project is created, you must enable the Cloud Storage Service for your project. must connect to the https://console.cloud.google.com/[Google Cloud Platform Console]
and create a new project. After your project is created, you must enable the
Cloud Storage Service for your project.
[[repository-gcs-creating-bucket]] [[repository-gcs-creating-bucket]]
===== Creating a Bucket ===== Creating a Bucket
Google Cloud Storage service uses the concept of https://cloud.google.com/storage/docs/key-terms[Bucket] The Google Cloud Storage service uses the concept of a
as a container for all the data. Buckets are usually created using the https://cloud.google.com/storage/docs/key-terms[bucket] as a container for all
https://console.cloud.google.com/[Google Cloud Platform Console]. The plugin will not automatically the data. Buckets are usually created using the
create buckets. https://console.cloud.google.com/[Google Cloud Platform Console]. The plugin
does not automatically create buckets.
To create a new bucket: To create a new bucket:
1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console] 1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console].
2. Select your project 2. Select your project.
3. Go to the https://console.cloud.google.com/storage/browser[Storage Browser] 3. Go to the https://console.cloud.google.com/storage/browser[Storage Browser].
4. Click the "Create Bucket" button 4. Click the *Create Bucket* button.
5. Enter the name of the new bucket 5. Enter the name of the new bucket.
6. Select a storage class 6. Select a storage class.
7. Select a location 7. Select a location.
8. Click the "Create" button 8. Click the *Create* button.
The bucket should now be created. For more detailed instructions, see the
https://cloud.google.com/storage/docs/quickstart-console#create_a_bucket[Google Cloud documentation].
[[repository-gcs-service-authentication]] [[repository-gcs-service-authentication]]
===== Service Authentication ===== Service Authentication
The plugin supports two authentication modes: The plugin must authenticate the requests it makes to the Google Cloud Storage
service. It is common for Google client libraries to employ a strategy named https://cloud.google.com/docs/authentication/production#providing_credentials_to_your_application[application default credentials].
* The built-in <<repository-gcs-using-compute-engine, Compute Engine authentication>>. This mode is However, that strategy is **not** supported for use with Elasticsearch. The
recommended if your Elasticsearch node is running on a Compute Engine virtual machine. plugin operates under the Elasticsearch process, which runs with the security
manager enabled. The security manager obstructs the "automatic" credential discovery.
* Specifying <<repository-gcs-using-service-account, Service Account>> credentials. Therefore, you must configure <<repository-gcs-using-service-account,service account>>
credentials even if you are using an environment that does not normally require
[[repository-gcs-using-compute-engine]] this configuration (such as Compute Engine, Kubernetes Engine or App Engine).
===== Using Compute Engine
When running on Compute Engine, the plugin use Google's built-in authentication mechanism to
authenticate on the Storage service. Compute Engine virtual machines are usually associated to a
default service account. This service account can be found in the VM instance details in the
https://console.cloud.google.com/compute/[Compute Engine console].
This is the default authentication mode and requires no configuration.
NOTE: The Compute Engine VM must be allowed to use the Storage service. This can be done only at VM
creation time, when "Storage" access can be configured to "Read/Write" permission. Check your
instance details at the section "Cloud API access scopes".
[[repository-gcs-using-service-account]] [[repository-gcs-using-service-account]]
===== Using a Service Account ===== Using a Service Account
If your Elasticsearch node is not running on Compute Engine, or if you don't want to use Google's You have to obtain and provide https://cloud.google.com/iam/docs/overview#service_account[service account credentials]
built-in authentication mechanism, you can authenticate on the Storage service using a manually.
https://cloud.google.com/iam/docs/overview#service_account[Service Account] file.
To create a service account file: For detailed information about generating JSON service account files, see the https://cloud.google.com/storage/docs/authentication?hl=en#service_accounts[Google Cloud documentation].
Note that the PKCS12 format is not supported by this plugin.
1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console] Here is a summary of the steps:
2. Select your project
3. Got to the https://console.cloud.google.com/permissions[Permission] tab
4. Select the https://console.cloud.google.com/permissions/serviceaccounts[Service Accounts] tab
5. Click on "Create service account"
6. Once created, select the new service account and download a JSON key file
A service account file looks like this: 1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console].
2. Select your project.
3. Got to the https://console.cloud.google.com/permissions[Permission] tab.
4. Select the https://console.cloud.google.com/permissions/serviceaccounts[Service Accounts] tab.
5. Click *Create service account*.
6. After the account is created, select it and download a JSON key file.
A JSON service account file looks like this:
[source,js] [source,js]
---- ----
@ -84,19 +79,26 @@ A service account file looks like this:
"private_key_id": "...", "private_key_id": "...",
"private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n", "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
"client_email": "service-account-for-your-repository@your-project-id.iam.gserviceaccount.com", "client_email": "service-account-for-your-repository@your-project-id.iam.gserviceaccount.com",
"client_id": "..." "client_id": "...",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/your-bucket@your-project-id.iam.gserviceaccount.com"
} }
---- ----
// NOTCONSOLE // NOTCONSOLE
This file must be stored in the {ref}/secure-settings.html[elasticsearch keystore], under a setting name To provide this file to the plugin, it must be stored in the {ref}/secure-settings.html[Elasticsearch keystore]. You must add a setting name of the form `gcs.client.NAME.credentials_file`, where `NAME`
of the form `gcs.client.NAME.credentials_file`, where `NAME` is the name of the client configuration. is the name of the client configuration for the repository. The implicit client
The default client name is `default`, but a different client name can be specified in repository name is `default`, but a different client name can be specified in the
settings using `client`. repository settings with the `client` key.
For example, if specifying the credentials file in the keystore under NOTE: Passing the file path via the GOOGLE_APPLICATION_CREDENTIALS environment
`gcs.client.my_alternate_client.credentials_file`, you can configure a repository to use these variable is **not** supported.
credentials like this:
For example, if you added a `gcs.client.my_alternate_client.credentials_file`
setting in the keystore, you can configure a repository to use those credentials
like this:
[source,js] [source,js]
---- ----
@ -113,19 +115,18 @@ PUT _snapshot/my_gcs_repository
// TEST[skip:we don't have gcs setup while testing this] // TEST[skip:we don't have gcs setup while testing this]
The `credentials_file` settings are {ref}/secure-settings.html#reloadable-secure-settings[reloadable]. The `credentials_file` settings are {ref}/secure-settings.html#reloadable-secure-settings[reloadable].
After you reload the settings, the internal `gcs` clients, used to transfer the After you reload the settings, the internal `gcs` clients, which are used to
snapshot contents, will utilize the latest settings from the keystore. transfer the snapshot contents, utilize the latest settings from the keystore.
NOTE: Snapshot or restore jobs that are in progress are not preempted by a *reload*
NOTE: In progress snapshot/restore jobs will not be preempted by a *reload* of the client's `credentials_file` settings. They complete using the client as
of the client's `credentials_file` settings. They will complete using the client it was built when the operation started.
as it was built when the operation started.
[[repository-gcs-client]] [[repository-gcs-client]]
==== Client Settings ==== Client Settings
The client used to connect to Google Cloud Storage has a number of settings available. The client used to connect to Google Cloud Storage has a number of settings available.
Client setting names are of the form `gcs.client.CLIENT_NAME.SETTING_NAME` and specified Client setting names are of the form `gcs.client.CLIENT_NAME.SETTING_NAME` and are specified
inside `elasticsearch.yml`. The default client name looked up by a `gcs` repository is inside `elasticsearch.yml`. The default client name looked up by a `gcs` repository is
called `default`, but can be customized with the repository setting `client`. called `default`, but can be customized with the repository setting `client`.
@ -146,7 +147,7 @@ PUT _snapshot/my_gcs_repository
// TEST[skip:we don't have gcs setup while testing this] // TEST[skip:we don't have gcs setup while testing this]
Some settings are sensitive and must be stored in the Some settings are sensitive and must be stored in the
{ref}/secure-settings.html[elasticsearch keystore]. This is the case for the service account file: {ref}/secure-settings.html[Elasticsearch keystore]. This is the case for the service account file:
[source,sh] [source,sh]
---- ----
@ -185,7 +186,7 @@ are marked as `Secure`.
`project_id`:: `project_id`::
The Google Cloud project id. This will be automatically infered from the credentials file but The Google Cloud project id. This will be automatically inferred from the credentials file but
can be specified explicitly. For example, it can be used to switch between projects when the can be specified explicitly. For example, it can be used to switch between projects when the
same credentials are usable for both the production and the development projects. same credentials are usable for both the production and the development projects.
@ -248,8 +249,8 @@ The following settings are supported:
The service account used to access the bucket must have the "Writer" access to the bucket: The service account used to access the bucket must have the "Writer" access to the bucket:
1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console] 1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console].
2. Select your project 2. Select your project.
3. Got to the https://console.cloud.google.com/storage/browser[Storage Browser] 3. Got to the https://console.cloud.google.com/storage/browser[Storage Browser].
4. Select the bucket and "Edit bucket permission" 4. Select the bucket and "Edit bucket permission".
5. The service account must be configured as a "User" with "Writer" access 5. The service account must be configured as a "User" with "Writer" access.