diff --git a/docs/development/extensions-core/kafka-ingestion.md b/docs/development/extensions-core/kafka-ingestion.md index 03553893a83..ad3c9c04190 100644 --- a/docs/development/extensions-core/kafka-ingestion.md +++ b/docs/development/extensions-core/kafka-ingestion.md @@ -221,15 +221,35 @@ The following example demonstrates supervisor spec with `lagBased` autoScaler en #### More on consumerProperties -This must contain a property `bootstrap.servers` with a list of Kafka brokers in the form: `:,:,...`. -By default, `isolation.level` is set to `read_committed`. It should be set to `read_uncommitted` if you don't want Druid to consume only committed transactions or working with older versions of Kafka servers with no transactions support. +Consumer properties must contain a property `bootstrap.servers` with a list of Kafka brokers in the form: `:,:,...`. +By default, `isolation.level` is set to `read_committed`. If you use older versions of Kafka servers without transactions support or don't want Druid to consume only committed transactions, set `isolation.level` to `read_uncommitted`. -There are few cases that require fetching few/all of consumer properties at runtime e.g. when `bootstrap.servers` is not known upfront or not static, to enable SSL connections users might have to provide passwords for `keystore`, `truststore` and `key` secretly. -For such consumer properties, user can implement a [DynamicConfigProvider](../../operations/dynamic-config-provider.md) to supply them at runtime, by adding -`druid.dynamic.config.provider`=`{"type": "", ...}` -in consumerProperties map. +In some cases, you may need to fetch consumer properties at runtime. For example, when `bootstrap.servers` is not known upfront, or is not static. To enable SSL connections, you must provide passwords for `keystore`, `truststore` and `key` secretly. You can provide configurations at runtime with a dynamic config provider implementation like the environment variable config provider that comes with Druid. For more information, see [DynamicConfigProvider](../../operations/dynamic-config-provider.md). -Note: SSL connections may also be supplied using the deprecated [Password Provider](../../operations/password-provider.md) interface to define the `keystore`, `truststore`, and `key`. This functionality might be removed in a future release. +For example, if you are using SASL and SSL with Kafka, set the following environment variables for the Druid user on the machines running the Overlord and the Peon services: + +``` +export KAFKA_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='admin_user' password='admin_password';" +export SSL_KEY_PASSWORD=mysecretkeypassword +export SSL_KEYSTORE_PASSWORD=mysecretkeystorepassword +export SSL_TRUSTSTORE_PASSWORD=mysecrettruststorepassword +``` + +``` + "druid.dynamic.config.provider": { + "type": "environment", + "variables": { + "sasl.jaas.config": "KAFKA_JAAS_CONFIG" + "ssl.key.password": "SSL_KEY_PASSWORD", + "ssl.keystore.password": "SSL_KEYSTORE_PASSWORD", + "ssl.truststore.password": "SSL_TRUSTSTORE_PASSWORD" + } + } + } +``` +Verify that you've changed the values for all configurations to match your own environment. You can use the environment variable config provider syntax in the **Consumer properties** field on the **Connect tab** in the **Load Data** UI in the Druid console. When connecting to Kafka, Druid replaces the environment variables with their corresponding values. + +Note: You can provide SSL connections with [Password Provider](../../operations/password-provider.md) interface to define the `keystore`, `truststore`, and `key`, but this feature is deprecated. #### Specifying data format diff --git a/docs/operations/dynamic-config-provider.md b/docs/operations/dynamic-config-provider.md index 45b61d54b94..0b34338562f 100644 --- a/docs/operations/dynamic-config-provider.md +++ b/docs/operations/dynamic-config-provider.md @@ -22,25 +22,58 @@ title: "Dynamic Config Providers" ~ under the License. --> -Druid's core mechanism of supplying multiple related set of credentials/secrets/configurations via Druid extension mechanism. Currently, it is only supported for providing Kafka Consumer configuration in [Kafka Ingestion](../development/extensions-core/kafka-ingestion.md). +Druid relies on dynamic config providers to supply multiple related sets of credentials, secrets, and configurations within a Druid extension. Dynamic config providers are intended to eventually replace [PasswordProvider](./password-provider.md). -Eventually this will replace [PasswordProvider](./password-provider.md) +By default, Druid includes an environment variable dynamic config provider that supports Kafka consumer configuration in [Kafka ingestion](../development/extensions-core/kafka-ingestion.md). +- Kafka consumer configuration in [Kafka ingestion](../development/extensions-core/kafka-ingestion.md) - -Users can create custom extension of the `DynamicConfigProvider` interface that is registered at Druid process startup. - -For more information, see [Adding a new DynamicConfigProvider implementation](../development/modules.md#adding-a-new-dynamicconfigprovider-implementation). +To develop a custom extension of the `DynamicConfigProvider` interface that is registered at Druid process startup, see [Adding a new DynamicConfigProvider implementation](../development/modules.md#adding-a-new-dynamicconfigprovider-implementation). ## Environment variable dynamic config provider -`EnvironmentVariableDynamicConfigProvider` can be used to avoid exposing credentials or other secret information in the configuration files using environment variables. An example to use this `configProvider` is: +You can use the environment variable dynamic config provider (`EnvironmentVariableDynamicConfigProvider`) to store passwords or other sensitive information using system environment variables instead of plain text configuration. + +The environment variable dynamic config provider uses the following syntax: + ```json -druid.some.config.dynamicConfigProvider={"type": "environment","variables":{"secret1": "SECRET1_VAR","secret2": "SECRET2_VAR"}} +druid.dynamic.config.provider={"type": "environment","variables":{"secret1": "SECRET1_VAR","secret2": "SECRET2_VAR"}} ``` -The values are described below. |Field|Type|Description|Required| |-----|----|-----------|--------| |`type`|String|dynamic config provider type|Yes: `environment`| -|`variables`|Map|environment variables to get information from|Yes| +|`variables`|Map|environment variables that store the configuration information|Yes| +When using the environment variable config provider, consider the following: +- If you manually specify a configuration key-value pair and use the dynamic config provider for the same key, Druid uses the value from the dynamic config provider. +- For use in a supervisor spec, environment variables must be available to the system user that runs the Overlord service and that runs the Peon service. + +The following example shows how to configure environment variables to store the SSL key and truststore passwords for Kafka. + +On the Overlord and Peon machines, set the following environment variables for the system user that runs the Druid services: + +``` +export SSL_KEY_PASSWORD=mysecretkeypassword +export SSL_KEYSTORE_PASSWORD=mysecretkeystorepassword +export SSL_TRUSTSTORE_PASSWORD=mysecrettruststorepassword +``` + +When you define the consumer properties in the supervisor spec, use the dynamic config provider to refer to the environment variables: +``` +... + "consumerProperties": { + "bootstrap.servers": "localhost:9092", + "ssl.keystore.location": "/opt/kafka/config/kafka01.keystore.jks" + "ssl.truststore.location": "/opt/kafka/config/kafka.truststore.jks" + "druid.dynamic.config.provider": { + "type": "environment", + "variables": { + "ssl.key.password": "SSL_KEY_PASSWORD", + "ssl.keystore.password": "SSL_KEYSTORE_PASSWORD", + "ssl.truststore.password": "SSL_TRUSTSTORE_PASSWORD" + } + } + }, +... +``` +When connecting to Kafka, Druid replaces the environment variables with their corresponding values. \ No newline at end of file diff --git a/website/.spelling b/website/.spelling index 705cc778269..182bf1eeec1 100644 --- a/website/.spelling +++ b/website/.spelling @@ -781,6 +781,7 @@ PT30M PT30S PT5S PT80S +SASL SegmentWriteOutMediumFactory UNABLE_TO_CONNECT_TO_STREAM UNHEALTHY_SUPERVISOR