mirror of https://github.com/apache/druid.git
Docs - update dynamic config provider topic (#11795)
* update dynamic config provider * update topic * add examples for dynamic config provider: * Update docs/development/extensions-core/kafka-ingestion.md Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com> * Update docs/development/extensions-core/kafka-ingestion.md Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com> * Update docs/development/extensions-core/kafka-ingestion.md Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com> * Update docs/operations/dynamic-config-provider.md Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com> * Update docs/operations/dynamic-config-provider.md Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com> * Update docs/operations/dynamic-config-provider.md Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com> * Update docs/operations/dynamic-config-provider.md Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com> * Update docs/development/extensions-core/kafka-ingestion.md Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com> * Update docs/operations/dynamic-config-provider.md Co-authored-by: Clint Wylie <cjwylie@gmail.com> * Update docs/operations/dynamic-config-provider.md Co-authored-by: Clint Wylie <cjwylie@gmail.com> * Update kafka-ingestion.md Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com> Co-authored-by: Clint Wylie <cjwylie@gmail.com>
This commit is contained in:
Charles Smith
GitHub
parent
4f62905be0
commit
6089a168ea
|
|
@ -221,15 +221,35 @@ The following example demonstrates supervisor spec with `lagBased` autoScaler en
|
|||
|
||||
#### More on consumerProperties
|
||||
|
||||
This must contain a property `bootstrap.servers` with a list of Kafka brokers in the form: `<BROKER_1>:<PORT_1>,<BROKER_2>:<PORT_2>,...`.
|
||||
By default, `isolation.level` is set to `read_committed`. It should be set to `read_uncommitted` if you don't want Druid to consume only committed transactions or working with older versions of Kafka servers with no transactions support.
|
||||
Consumer properties must contain a property `bootstrap.servers` with a list of Kafka brokers in the form: `<BROKER_1>:<PORT_1>,<BROKER_2>:<PORT_2>,...`.
|
||||
By default, `isolation.level` is set to `read_committed`. If you use older versions of Kafka servers without transactions support or don't want Druid to consume only committed transactions, set `isolation.level` to `read_uncommitted`.
|
||||
|
||||
There are few cases that require fetching few/all of consumer properties at runtime e.g. when `bootstrap.servers` is not known upfront or not static, to enable SSL connections users might have to provide passwords for `keystore`, `truststore` and `key` secretly.
|
||||
For such consumer properties, user can implement a [DynamicConfigProvider](../../operations/dynamic-config-provider.md) to supply them at runtime, by adding
|
||||
`druid.dynamic.config.provider`=`{"type": "<registered_dynamic_config_provider_name>", ...}`
|
||||
in consumerProperties map.
|
||||
In some cases, you may need to fetch consumer properties at runtime. For example, when `bootstrap.servers` is not known upfront, or is not static. To enable SSL connections, you must provide passwords for `keystore`, `truststore` and `key` secretly. You can provide configurations at runtime with a dynamic config provider implementation like the environment variable config provider that comes with Druid. For more information, see [DynamicConfigProvider](../../operations/dynamic-config-provider.md).
|
||||
|
||||
Note: SSL connections may also be supplied using the deprecated [Password Provider](../../operations/password-provider.md) interface to define the `keystore`, `truststore`, and `key`. This functionality might be removed in a future release.
|
||||
For example, if you are using SASL and SSL with Kafka, set the following environment variables for the Druid user on the machines running the Overlord and the Peon services:
|
||||
|
||||
```
|
||||
export KAFKA_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='admin_user' password='admin_password';"
|
||||
export SSL_KEY_PASSWORD=mysecretkeypassword
|
||||
export SSL_KEYSTORE_PASSWORD=mysecretkeystorepassword
|
||||
export SSL_TRUSTSTORE_PASSWORD=mysecrettruststorepassword
|
||||
```
|
||||
|
||||
```
|
||||
"druid.dynamic.config.provider": {
|
||||
"type": "environment",
|
||||
"variables": {
|
||||
"sasl.jaas.config": "KAFKA_JAAS_CONFIG"
|
||||
"ssl.key.password": "SSL_KEY_PASSWORD",
|
||||
"ssl.keystore.password": "SSL_KEYSTORE_PASSWORD",
|
||||
"ssl.truststore.password": "SSL_TRUSTSTORE_PASSWORD"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Verify that you've changed the values for all configurations to match your own environment. You can use the environment variable config provider syntax in the **Consumer properties** field on the **Connect tab** in the **Load Data** UI in the Druid console. When connecting to Kafka, Druid replaces the environment variables with their corresponding values.
|
||||
|
||||
Note: You can provide SSL connections with [Password Provider](../../operations/password-provider.md) interface to define the `keystore`, `truststore`, and `key`, but this feature is deprecated.
|
||||
|
||||
#### Specifying data format
|
||||
|
||||
|
|
|
|||
|
|
@ -22,25 +22,58 @@ title: "Dynamic Config Providers"
|
|||
~ under the License.
|
||||
-->
|
||||
|
||||
Druid's core mechanism of supplying multiple related set of credentials/secrets/configurations via Druid extension mechanism. Currently, it is only supported for providing Kafka Consumer configuration in [Kafka Ingestion](../development/extensions-core/kafka-ingestion.md).
|
||||
Druid relies on dynamic config providers to supply multiple related sets of credentials, secrets, and configurations within a Druid extension. Dynamic config providers are intended to eventually replace [PasswordProvider](./password-provider.md).
|
||||
|
||||
Eventually this will replace [PasswordProvider](./password-provider.md)
|
||||
By default, Druid includes an environment variable dynamic config provider that supports Kafka consumer configuration in [Kafka ingestion](../development/extensions-core/kafka-ingestion.md).
|
||||
- Kafka consumer configuration in [Kafka ingestion](../development/extensions-core/kafka-ingestion.md)
|
||||
|
||||
|
||||
Users can create custom extension of the `DynamicConfigProvider` interface that is registered at Druid process startup.
|
||||
|
||||
For more information, see [Adding a new DynamicConfigProvider implementation](../development/modules.md#adding-a-new-dynamicconfigprovider-implementation).
|
||||
To develop a custom extension of the `DynamicConfigProvider` interface that is registered at Druid process startup, see [Adding a new DynamicConfigProvider implementation](../development/modules.md#adding-a-new-dynamicconfigprovider-implementation).
|
||||
|
||||
## Environment variable dynamic config provider
|
||||
|
||||
`EnvironmentVariableDynamicConfigProvider` can be used to avoid exposing credentials or other secret information in the configuration files using environment variables. An example to use this `configProvider` is:
|
||||
You can use the environment variable dynamic config provider (`EnvironmentVariableDynamicConfigProvider`) to store passwords or other sensitive information using system environment variables instead of plain text configuration.
|
||||
|
||||
The environment variable dynamic config provider uses the following syntax:
|
||||
|
||||
```json
|
||||
druid.some.config.dynamicConfigProvider={"type": "environment","variables":{"secret1": "SECRET1_VAR","secret2": "SECRET2_VAR"}}
|
||||
druid.dynamic.config.provider={"type": "environment","variables":{"secret1": "SECRET1_VAR","secret2": "SECRET2_VAR"}}
|
||||
```
|
||||
The values are described below.
|
||||
|
||||
|Field|Type|Description|Required|
|
||||
|-----|----|-----------|--------|
|
||||
|`type`|String|dynamic config provider type|Yes: `environment`|
|
||||
|`variables`|Map|environment variables to get information from|Yes|
|
||||
|`variables`|Map|environment variables that store the configuration information|Yes|
|
||||
|
||||
When using the environment variable config provider, consider the following:
|
||||
- If you manually specify a configuration key-value pair and use the dynamic config provider for the same key, Druid uses the value from the dynamic config provider.
|
||||
- For use in a supervisor spec, environment variables must be available to the system user that runs the Overlord service and that runs the Peon service.
|
||||
|
||||
The following example shows how to configure environment variables to store the SSL key and truststore passwords for Kafka.
|
||||
|
||||
On the Overlord and Peon machines, set the following environment variables for the system user that runs the Druid services:
|
||||
|
||||
```
|
||||
export SSL_KEY_PASSWORD=mysecretkeypassword
|
||||
export SSL_KEYSTORE_PASSWORD=mysecretkeystorepassword
|
||||
export SSL_TRUSTSTORE_PASSWORD=mysecrettruststorepassword
|
||||
```
|
||||
|
||||
When you define the consumer properties in the supervisor spec, use the dynamic config provider to refer to the environment variables:
|
||||
```
|
||||
...
|
||||
"consumerProperties": {
|
||||
"bootstrap.servers": "localhost:9092",
|
||||
"ssl.keystore.location": "/opt/kafka/config/kafka01.keystore.jks"
|
||||
"ssl.truststore.location": "/opt/kafka/config/kafka.truststore.jks"
|
||||
"druid.dynamic.config.provider": {
|
||||
"type": "environment",
|
||||
"variables": {
|
||||
"ssl.key.password": "SSL_KEY_PASSWORD",
|
||||
"ssl.keystore.password": "SSL_KEYSTORE_PASSWORD",
|
||||
"ssl.truststore.password": "SSL_TRUSTSTORE_PASSWORD"
|
||||
}
|
||||
}
|
||||
},
|
||||
...
|
||||
```
|
||||
When connecting to Kafka, Druid replaces the environment variables with their corresponding values.
|
||||
|
|
@ -781,6 +781,7 @@ PT30M
|
|||
PT30S
|
||||
PT5S
|
||||
PT80S
|
||||
SASL
|
||||
SegmentWriteOutMediumFactory
|
||||
UNABLE_TO_CONNECT_TO_STREAM
|
||||
UNHEALTHY_SUPERVISOR
|
||||
|
|
|
|||
Loading…
Reference in New Issue