From bd899b922459e36ff1fba883da068f8d415cbf47 Mon Sep 17 00:00:00 2001 From: gocho1 Date: Wed, 22 May 2019 20:46:02 +0200 Subject: [PATCH] add s3 authentication method informations (#7674) * add s3 authentication method informations * add druid.s3.fileSessionCredentials related content * remove authentication parameters to avoid confusion as it is more detailed in S3 Deep Storage page * streamline s3 docs --- docs/content/configuration/index.md | 5 ++- .../content/development/extensions-core/s3.md | 39 +++++++++++++++---- 2 files changed, 34 insertions(+), 10 deletions(-) diff --git a/docs/content/configuration/index.md b/docs/content/configuration/index.md index 29c11fa05c5..25e0ed936ab 100644 --- a/docs/content/configuration/index.md +++ b/docs/content/configuration/index.md @@ -541,13 +541,14 @@ The below table shows some important configurations for S3. See [S3 Deep Storage |Property|Description|Default| |--------|-----------|-------| -|`druid.s3.accessKey`|The access key to use to access S3.|none| -|`druid.s3.secretKey`|The secret key to use to access S3.|none| |`druid.storage.bucket`|S3 bucket name.|none| |`druid.storage.baseKey`|S3 object key prefix for storage.|none| |`druid.storage.disableAcl`|Boolean flag for ACL. If this is set to `false`, the full control would be granted to the bucket owner. This may require to set additional permissions. See [S3 permissions settings](../development/extensions-core/s3.html#s3-permissions-settings).|false| |`druid.storage.archiveBucket`|S3 bucket name for archiving when running the *archive task*.|none| |`druid.storage.archiveBaseKey`|S3 object key prefix for archiving.|none| +|`druid.storage.sse.type`|Server-side encryption type. Should be one of `s3`, `kms`, and `custom`. See the below [Server-side encryption section](../development/extensions-core/s3.html#server-side-encryption) for more details.|None| +|`druid.storage.sse.kms.keyId`|AWS KMS key ID. This is used only when `druid.storage.sse.type` is `kms` and can be empty to use the default key ID.|None| +|`druid.storage.sse.custom.base64EncodedKey`|Base64-encoded key. Should be specified if `druid.storage.sse.type` is `custom`.|None| |`druid.storage.useS3aSchema`|If true, use the "s3a" filesystem when using Hadoop-based ingestion. If false, the "s3n" filesystem will be used. Only affects Hadoop-based ingestion.|false| #### HDFS Deep Storage diff --git a/docs/content/development/extensions-core/s3.md b/docs/content/development/extensions-core/s3.md index 2fa18296180..41b4b565715 100644 --- a/docs/content/development/extensions-core/s3.md +++ b/docs/content/development/extensions-core/s3.md @@ -41,14 +41,9 @@ As an example, to set the region to 'us-east-1' through system properties: |Property|Description|Default| |--------|-----------|-------| -|`druid.s3.accessKey`|S3 access key.|Must be set.| -|`druid.s3.secretKey`|S3 secret key.|Must be set.| -|`druid.storage.bucket`|Bucket to store in.|Must be set.| -|`druid.storage.baseKey`|Base key prefix to use, i.e. what directory.|Must be set.| -|`druid.storage.disableAcl`|Boolean flag to disable ACL. If this is set to `false`, the full control would be granted to the bucket owner. This may require to set additional permissions. See [S3 permissions settings](#s3-permissions-settings).|false| -|`druid.storage.sse.type`|Server-side encryption type. Should be one of `s3`, `kms`, and `custom`. See the below [Server-side encryption section](#server-side-encryption) for more details.|None| -|`druid.storage.sse.kms.keyId`|AWS KMS key ID. This is used only when `druid.storage.sse.type` is `kms` and can be empty to use the default key ID.|None| -|`druid.storage.sse.custom.base64EncodedKey`|Base64-encoded key. Should be specified if `druid.storage.sse.type` is `custom`.|None| +|`druid.s3.accessKey`|S3 access key.See [S3 authentication methods](#s3-authentication-methods) for more details|Can be ommitted according to authentication methods chosen.| +|`druid.s3.secretKey`|S3 secret key.See [S3 authentication methods](#s3-authentication-methods) for more details|Can be ommitted according to authentication methods chosen.| +|`druid.s3.fileSessionCredentials`|Path to properties file containing `sessionToken`, `accessKey` and `secretKey` value. One key/value pair per line (format `key=value`). See [S3 authentication methods](#s3-authentication-methods) for more details |Can be ommitted according to authentication methods chosen.| |`druid.s3.protocol`|Communication protocol type to use when sending requests to AWS. `http` or `https` can be used. This configuration would be ignored if `druid.s3.endpoint.url` is filled with a URL with a different protocol.|`https`| |`druid.s3.disableChunkedEncoding`|Disables chunked encoding. See [AWS document](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Builder.html#disableChunkedEncoding--) for details.|false| |`druid.s3.enablePathStyleAccess`|Enables path style access. See [AWS document](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Builder.html#enablePathStyleAccess--) for details.|false| @@ -59,12 +54,38 @@ As an example, to set the region to 'us-east-1' through system properties: |`druid.s3.proxy.port`|Port on the proxy host to connect through.|None| |`druid.s3.proxy.username`|User name to use when connecting through a proxy.|None| |`druid.s3.proxy.password`|Password to use when connecting through a proxy.|None| +|`druid.storage.bucket`|Bucket to store in.|Must be set.| +|`druid.storage.baseKey`|Base key prefix to use, i.e. what directory.|Must be set.| +|`druid.storage.archiveBucket`|S3 bucket name for archiving when running the *archive task*.|none| +|`druid.storage.archiveBaseKey`|S3 object key prefix for archiving.|none| +|`druid.storage.disableAcl`|Boolean flag to disable ACL. If this is set to `false`, the full control would be granted to the bucket owner. This may require to set additional permissions. See [S3 permissions settings](#s3-permissions-settings).|false| +|`druid.storage.sse.type`|Server-side encryption type. Should be one of `s3`, `kms`, and `custom`. See the below [Server-side encryption section](#server-side-encryption) for more details.|None| +|`druid.storage.sse.kms.keyId`|AWS KMS key ID. This is used only when `druid.storage.sse.type` is `kms` and can be empty to use the default key ID.|None| +|`druid.storage.sse.custom.base64EncodedKey`|Base64-encoded key. Should be specified if `druid.storage.sse.type` is `custom`.|None| +|`druid.storage.useS3aSchema`|If true, use the "s3a" filesystem when using Hadoop-based ingestion. If false, the "s3n" filesystem will be used. Only affects Hadoop-based ingestion.|false| ### S3 permissions settings `s3:GetObject` and `s3:PutObject` are basically required for pushing/loading segments to/from S3. If `druid.storage.disableAcl` is set to `false`, then `s3:GetBucketAcl` and `s3:PutObjectAcl` are additionally required to set ACL for objects. +### S3 authentication methods + +To connect to your S3 bucket (whether deep storage bucket or source bucket), Druid use the following credentials providers chain + +|order|type|details| +|--------|-----------|-------| +|1|Druid config file|Based on your runtime.properties if it contains values `druid.s3.accessKey` and `druid.s3.secretKey` | +|2|Custom properties file| Based on custom properties file where you can supply `sessionToken`, `accessKey` and `secretKey` values. This file is provided to Druid through `druid.s3.fileSessionCredentials` propertie| +|3|Environment variables|Based on environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`| +|4|Java system properties|Based on JVM properties `aws.accessKeyId` and `aws.secretKey` | +|5|Profile informations|Based on credentials you may have on your druid instance (generally in `~/.aws/credentials`)| +|6|Instance profile informations|Based on the instance profile you may have attached to your druid instance| + +You can find more informations about authentication method [here](https://docs.aws.amazon.com/fr_fr/sdk-for-java/v1/developer-guide/credentials.html)
+**Note :** *Order is important here as it indicates the precedence of authentication methods.
+So if you are trying to use Instance profile informations, you **must not** set `druid.s3.accessKey` and `druid.s3.secretKey` in your Druid runtime.properties* + ## Server-side encryption You can enable [server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html) by setting @@ -102,3 +123,5 @@ shardSpecs are not specified, and, in this case, caching can be useful. Prefetch |prefetchTriggerBytes|Threshold to trigger prefetching s3 objects.|maxFetchCapacityBytes / 2|no| |fetchTimeout|Timeout for fetching an s3 object.|60000|no| |maxFetchRetry|Maximum retry for fetching an s3 object.|3|no| + +