mirror of https://github.com/apache/druid.git
Cleanup the base lookup cluster wide config docs (#3061)
* Cleanup the base lookup cluster wide config docs * Add better examples in lookups-cached-global.md * Use actual valid stock lookups * Fixed maps with : * Add mix of lookups * Better examples in extension * Remove unneeded namespace requirement * Add extra line space * Add link to lookup tiers * Renamed header
This commit is contained in:
Charles Allen
Fangjin Yang
parent
e6b32c24ae
commit
fa41a6466a
|
|
@ -28,7 +28,6 @@ Cached namespace lookups can be specified as part of the [cluster wide config fo
|
|||
"type": "cachedNamespace",
|
||||
"extractionNamespace": {
|
||||
"type": "uri",
|
||||
"namespace": "some_uri_lookup",
|
||||
"uri": "file:/tmp/prefix/",
|
||||
"namespaceParseSpec": {
|
||||
"format": "csv",
|
||||
|
|
@ -48,7 +47,6 @@ Cached namespace lookups can be specified as part of the [cluster wide config fo
|
|||
"type": "cachedNamespace",
|
||||
"extractionNamespace": {
|
||||
"type": "jdbc",
|
||||
"namespace": "some_jdbc_lookup",
|
||||
"connectorConfig": {
|
||||
"createTables": true,
|
||||
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
|
||||
|
|
@ -78,6 +76,58 @@ If `firstCacheTimeout` is set to a non-zero value, it should be less than `druid
|
|||
Proper functionality of Namespaced lookups requires the following extension to be loaded on the broker, peon, and historical nodes:
|
||||
`druid-lookups-cached-global`
|
||||
|
||||
## Example configuration
|
||||
|
||||
In a simple case where only one [tier](../../querying/lookups.html#dynamic-configuration) exists (`realtime_customer2`) with one `cachedNamespace` lookup called `country_code`, the resulting configuration json looks similar to the following:
|
||||
|
||||
```json
|
||||
{
|
||||
"realtime_customer2": {
|
||||
"country_code": {
|
||||
"type": "cachedNamespace",
|
||||
"extractionNamespace": {
|
||||
"type": "jdbc",
|
||||
"connectorConfig": {
|
||||
"createTables": true,
|
||||
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
|
||||
"user": "druid",
|
||||
"password": "diurd"
|
||||
},
|
||||
"table": "lookupTable",
|
||||
"keyColumn": "country_id",
|
||||
"valueColumn": "country_name",
|
||||
"tsColumn": "timeColumn"
|
||||
},
|
||||
"firstCacheTimeout": 120000,
|
||||
"injective":true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Where the coordinator endpoint `/druid/coordinator/v1/lookups/realtime_customer2/country_code` should return
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "cachedNamespace",
|
||||
"extractionNamespace": {
|
||||
"type": "jdbc",
|
||||
"connectorConfig": {
|
||||
"createTables": true,
|
||||
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
|
||||
"user": "druid",
|
||||
"password": "diurd"
|
||||
},
|
||||
"table": "lookupTable",
|
||||
"keyColumn": "country_id",
|
||||
"valueColumn": "country_name",
|
||||
"tsColumn": "timeColumn"
|
||||
},
|
||||
"firstCacheTimeout": 120000,
|
||||
"injective":true
|
||||
}
|
||||
```
|
||||
|
||||
## Cache Settings
|
||||
|
||||
Lookups are cached locally on historical nodes. The following are settings used by the nodes which service queries when
|
||||
|
|
@ -126,9 +176,9 @@ The remapping values for each namespaced lookup can be specified by a json objec
|
|||
"pollPeriod":"PT5M"
|
||||
}
|
||||
```
|
||||
|
||||
|Property|Description|Required|Default|
|
||||
|--------|-----------|--------|-------|
|
||||
|`namespace`|The namespace to define|Yes||
|
||||
|`pollPeriod`|Period between polling for updates|No|0 (only once)|
|
||||
|`uri`|URI for the file of interest|No|Use `uriPrefix`|
|
||||
|`uriPrefix`|A URI which specifies a directory (or other searchable resource) in which to search for files|No|Use `uri`|
|
||||
|
|
|
|||
|
|
@ -79,6 +79,7 @@ Lookups can be updated in bulk by posting a JSON object to `/druid/coordinator/v
|
|||
{
|
||||
"tierName": {
|
||||
"lookupExtractorFactoryName": {
|
||||
"type": "someExtractorFactoryType",
|
||||
"someExtractorField": "someExtractorValue"
|
||||
}
|
||||
}
|
||||
|
|
@ -91,55 +92,54 @@ So a config might look something like:
|
|||
{
|
||||
"__default": {
|
||||
"country_code": {
|
||||
"type": "simple_json",
|
||||
"uri": "http://some.host.com/codes.json"
|
||||
"type": "map",
|
||||
"map": {"77483": "United States"}
|
||||
},
|
||||
"site_id": {
|
||||
"type": "confidential_jdbc",
|
||||
"auth": "/etc/jdbc.internal",
|
||||
"table": "sites",
|
||||
"key": "site_id",
|
||||
"value": "site_name"
|
||||
"type": "cachedNamespace",
|
||||
"extractionNamespace": {
|
||||
"type": "jdbc",
|
||||
"connectorConfig": {
|
||||
"createTables": true,
|
||||
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
|
||||
"user": "druid",
|
||||
"password": "diurd"
|
||||
},
|
||||
"table": "lookupTable",
|
||||
"keyColumn": "country_id",
|
||||
"valueColumn": "country_name",
|
||||
"tsColumn": "timeColumn"
|
||||
},
|
||||
"firstCacheTimeout": 120000,
|
||||
"injective":true
|
||||
},
|
||||
"site_id_customer1": {
|
||||
"type": "confidential_jdbc",
|
||||
"auth": "/etc/jdbc.customer1",
|
||||
"table": "sites",
|
||||
"key": "site_id",
|
||||
"value": "site_name"
|
||||
"type": "map",
|
||||
"map": {"847632": "Internal Use Only"}
|
||||
},
|
||||
"site_id_customer2": {
|
||||
"type": "confidential_jdbc",
|
||||
"auth": "/etc/jdbc.customer2",
|
||||
"table": "sites",
|
||||
"key": "site_id",
|
||||
"value": "site_name"
|
||||
"type": "map",
|
||||
"map": {"AHF77": "Home"}
|
||||
}
|
||||
},
|
||||
"realtime_customer1": {
|
||||
"country_code": {
|
||||
"type": "simple_json",
|
||||
"uri": "http://some.host.com/codes.json"
|
||||
"type": "map",
|
||||
"map": {"77483": "United States"}
|
||||
},
|
||||
"site_id_customer1": {
|
||||
"type": "confidential_jdbc",
|
||||
"auth": "/etc/jdbc.customer1",
|
||||
"table": "sites",
|
||||
"key": "site_id",
|
||||
"value": "site_name"
|
||||
"type": "map",
|
||||
"map": {"847632": "Internal Use Only"}
|
||||
}
|
||||
},
|
||||
"realtime_customer2": {
|
||||
"country_code": {
|
||||
"type": "simple_json",
|
||||
"uri": "http://some.host.com/codes.json"
|
||||
"type": "map",
|
||||
"map": {"77483": "United States"}
|
||||
},
|
||||
"site_id_customer2": {
|
||||
"type": "confidential_jdbc",
|
||||
"auth": "/etc/jdbc.customer2",
|
||||
"table": "sites",
|
||||
"key": "site_id",
|
||||
"value": "site_name"
|
||||
"type": "map",
|
||||
"map": {"AHF77": "Home"}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -154,11 +154,8 @@ For example, a post to `/druid/coordinator/v1/lookups/realtime_customer1/site_id
|
|||
|
||||
```json
|
||||
{
|
||||
"type": "confidential_jdbc",
|
||||
"auth": "/etc/jdbc.customer1",
|
||||
"table": "sites_updated",
|
||||
"key": "site_id",
|
||||
"value": "site_name"
|
||||
"type": "map",
|
||||
"map": {"847632": "Internal Use Only"}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -171,11 +168,8 @@ Using the prior example, a `GET` to `/druid/coordinator/v1/lookups/realtime_cust
|
|||
|
||||
```json
|
||||
{
|
||||
"type": "confidential_jdbc",
|
||||
"auth": "/etc/jdbc.customer2",
|
||||
"table": "sites",
|
||||
"key": "site_id",
|
||||
"value": "site_name"
|
||||
"type": "map",
|
||||
"map": {"AHF77": "Home"}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -205,10 +199,10 @@ The return value will be a json map of the lookups to their extractor factories.
|
|||
```json
|
||||
|
||||
{
|
||||
"some_lookup_name": {
|
||||
"type": "simple_json",
|
||||
"uri": "http://some.host.com/codes.json"
|
||||
}
|
||||
"some_lookup_name": {
|
||||
"type": "map",
|
||||
"map": {"77483": "United States"}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
|
@ -220,8 +214,8 @@ The return value will be the json representation of the factory.
|
|||
|
||||
```json
|
||||
{
|
||||
"type": "simple_json",
|
||||
"uri": "http://some.host.com/codes.json"
|
||||
"type": "map",
|
||||
"map": {"77483", "United States"}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -232,8 +226,8 @@ The return value will be a JSON map in the following format:
|
|||
|
||||
```json
|
||||
{
|
||||
"status": "accepted",
|
||||
"failedUpdates": {}
|
||||
"status": "accepted",
|
||||
"failedUpdates": {}
|
||||
}
|
||||
|
||||
```
|
||||
|
|
@ -242,13 +236,13 @@ If a lookup cannot be started, or is left in an undefined state, the lookup in e
|
|||
|
||||
```json
|
||||
{
|
||||
"status": "accepted",
|
||||
"failedUpdates": {
|
||||
"country_code": {
|
||||
"type": "simple_json",
|
||||
"uri": "http://some.host.com/codes.json"
|
||||
}
|
||||
"status": "accepted",
|
||||
"failedUpdates": {
|
||||
"country_code": {
|
||||
"type": "map",
|
||||
"map": {"77483": "United States"}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
|
@ -263,8 +257,8 @@ If `some_lookup_name` is desired to have the LookupExtractorFactory definition o
|
|||
|
||||
```json
|
||||
{
|
||||
"type": "simple_json",
|
||||
"uri": "http://some.host.com/codes.json"
|
||||
"type": "map",
|
||||
"map": {"77483": "United States"}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -273,10 +267,10 @@ Then a post to `/druid/listen/v1/lookups/some_lookup_name` will behave the same
|
|||
```json
|
||||
|
||||
{
|
||||
"some_lookup_name": {
|
||||
"type": "simple_json",
|
||||
"uri": "http://some.host.com/codes.json"
|
||||
}
|
||||
"some_lookup_name": {
|
||||
"type": "map",
|
||||
"map": {"77483": "United States"}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
|
|
|||
Loading…
Reference in New Issue